compound.htm: Fravia's web-searching lore: metasearch engines

Madame Schmidt compounding & an interesting small seekers riddle: what are those two black balls?

~ Compound ("Meta") search engines ~
Updated March 2007

[Introduction]

[Compound searches] [Compound pointers]

Seekz (!)

[Mamma]

[Dogpile]

[Metacrawler]

[Vivisimo] (very good parsing!)

[Fazzle]

[Surfwax]

[Inquirus] (±)   [Ixquick]   [Clusty](ß) [Mearf](ß) [jux2](ß)
[eTools.ch]

[Snooz]   [Ilectric]   [Iboogie] [search.com] (±)
   [Ithaki]   [Debriefing (ex Ixquick)]   [Metor]   [Queryserver]

Regional Metasearches
DE: [Metager]

[Corrections and Additions] [Metacrawler form] ["Metameta" form]

Best compound engines: [Seekz] [Dogpile] [Vivisimo] [Fazzle]
[Mamma] [Surfwax] [Ez2find]

Introduction

"Metasearch" engines
(aka "compound" ~ "parallel" ~ "inference" engines, aka "pumps" aka "metas", aka "multi-threaded")

Metasearch engines are search tools that send a query simultaneously to several search engines and Web directories... and sometimes to the so-called Invisible (Deep) Web: online informationand databases not indexed by the traditional main search engines.

Advantages: they query more search engines at the same time, which is important given the low degree of overlapping among the main search engines.
Disadvantages: 1) they are good mostly only for unique terms queries; 2) they spend a short time in each database; 3) they discard complex searching logic; 4) they try to "accommodate" all search engines they drain using always the same query (not very good); 5) they don't drain some important main search engines (which you should always consider using if you want to cover more than 1/3 ~ 1/2 of the Web).

Alas! Many 'compound' (or 'meta' or 'parallel') search engines are NOT what they promise to be. They are just all-in-one simple CGI scripts, trying to attract traffic to their sites simply querying with the SAME querymask various main search engines, one after the other.

In order to check how really META yor metasearch engine is, you should take account of the following parameters:

the engine should perform a real PARALLEL search, not a sequential one, for obvious speed reasons
all results should be integrated, compounded and presented in a uniform format
all the doubles should be eliminated (telling you what happened)
the engine should offer you --at least-- the boolean operators AND and OR
the engine should perform a complete search (the metaengine should not 'cut' the search when the various search engines are still giving results)

As you probably know, there is a huge difference between the results that the main search engines pretend to have found and the results they will show you (see for some examples the [yoyo] searching technique). A good metasearch engine will always inform you about the claimed reported results and the effectively found results.

Our own scrolls

Internet searching is not a matter of just dropping one word inside google. In order to query at the same time more than one search engine, of course, you could use our "scrolls" - short php bots, prepared by Laurent with the help of DQ at the [PHP Lab]. The most obvious advantages -for you- would be the absolute absence of any advertisement whatsoever, the possibility to build on the source code and some fairly advanced configuration possibilities. Give them a try!

To the scrolls room!

Major scrolls:		Blue Scroll Of Searching Clear Scroll of Clairvoyance
Minor scrolls:		Indigo Scroll Of Local searchings Purple Scroll of Effective searching Azure Scroll Of Second Effective searching Lavender Scroll Of most recent engines

Compound searches

Inquirus, one of the most promising compound search engines...
Alas! now DEFUNCT: here some story.
There was still a working prototype, though: http://inquirus.nj.nec.com/i2/inq2.pl (see http://inquirus.nj.nec.com/).

" The Inquirus metasearch engine - real-time analysis of current web contents. Query-sensitive summaries and Specific Expressive Forms for question answering."

Ixquick uses Alta's style rules. Space search terms to search for pages with as many of the terms as possible. Use the + operator to compel.
AND, NEAR, Parentheses, etc.
Additionally you can specify where certain information must appear with fields. Currently supported fields include:
title: page title must contain this text
domain: page must be located at a .com or .gov or .edu or .de (etc.) computer
host: page must be located on this specified computer or on more specific computers (e.g. host:searchlores.org will retrieve pages on searchlores.org and on www.searchlores.org)
image: page must contain an image whose file name contains the specified text.
url: find pages with specified text anywhere within the url, e.g. url:de will find http://abc.org/pics/de.html as well as http://www.de/ link: page must contain a hyperlink for which at least one of the words in the destination address begins with specified text.
text: actual text displayed on the page must contain specified text (otherwise the text might appear in the title or in the hypertext code but not on the page)
related: find sites with similar content to the specified page

Clusty

Uses GigaBlast, MSN, Lycos, Looksmart, Wisenut, Open Directory & Overture
"Clusters" links... clicking on the name of a cluster will display all of the search results that it contains

Mearf

Uses Alltheweb, altavista, google & yahoo
Mearf is an experimental meta search engine using content based collection fusion methods. It sends the given query up to 5 different search engines, and merges the results using different collection fusion strategies.

Link info

alltheweb altavista google yahoo

http://jux2.com/

Uses google, yahoo and AskJeeves
Jux2 is a beta meta engine intended as a comparative research tool to check (mainly) the differences between google's, msn and yahoo's results. Jux2 claims that their three metapool search engines typically share fewer than 3.5 results among their respective top 10 results. Very interesting compare function.

search , and simultaneously

advanced search
preferences

http://www.etools.ch/mobileSearch.do

Stephan's GOOD meta-engine: quick, powerful, free.
"What is different compared to other metasearch engines?
The main features:
- search in different languages and European countries, not just worldwide
- parses the entered query and _really_ translates it to each target
engine (boolean operators, modifiers and phrases)
- allows you to examine the results of each source individually
- allows you to save the merged result as PDF- or RSS file
- one of the few metasearch engine who searches in Google
- allows you to weight or disable each search engine individually (see Preferences)
- removes results with tracking information (especially MSN, Yahoo, Altavista and Lycos contain randomly redirect links in their results)
- shows only max. 3 sponsored links that help me pay the hosting (max. 2 on Mobile Search)"

For instance http://www.etools.ch/searchSubmit.do?query=%22advanced+searching%22&country=web&language=all

Advanced search: http://www.etools.ch/searchAdvanced.do (european finetuning galore!)

How does it work?

THE BEST ONE!
is [seekz], without advertisements & crap
Seekz
(Maybe the best Metasearch engine)

Seekz is a parallel web search engine, or metasearcher. It checks 15 search sites in one go.
It is a university project and strictly non-commercial, so it excludes results from engines that accept payment for higher placements and ignores PPC results at the rest.
Its own pages are free from banners, pop-ups and other forms of advertising. As well as visting better known engines like MSN Search, Yahoo! and Alta Vista, Seekz also stitches together listings from second and third tier searchers, including Exalead, Gigablast and Wotbox.

[Mamma Metasearch]
http://www.mamma.com/psearch.html

For instance
http://www.mamma.com/MammaPSearch?query=advanced+searching&eng%5B%5D=1681&eng%5B%5D=1697&eng%5B%5D=1668&eng%5B%5D=1691&eng%5B%5D=1701&eng%5B%5D=1694&eng%5B%5D=1791&eng%5B%5D=1620&eng%5B%5D=19&highlight=1&timeout=3&rpp=15&desc=0&savesettings=0&qtype=0&utfout=1

DIRECTORIES:
Open Directory Looksmart Directory Business.com About.com Mamma's Collection

INDEXES:
Teoma Google MSN Entireweb Gigablast

Mamma supports advanced operators:
• Exact match using "quotations"
• Compelled terms are included using +plus +signs
• Undesired terms are excluded using the -minus -sign

Also note their "refine your search" options à la ex-teoma (now ask.com)

Canadian compound engine.
Here their own spin: "Created in 1996 as a master's thesis, Mamma.com helped to introduce metasearch to the Internet as one of the first of its kind. Due to its quality results, and the benefits of metasearch, Mamma grew rapidly through word of mouth, and quickly became an established search engine on the Internet"

[Snooz Metasearch]
http://www.ijs.co.nz/info/snooz.htm

Snooz Metasearch was a very good metasearch engine, it is now a tag obsolete.
Snooz Metasearch. also has syntax translation (e.g. it translates "&" to "AND" and "+link:" to "+linkdomain:" (for HotBot/Anzwers/etc.)). It allows use of Booleans and dates (e.g. in queries going to AltaVista), which are features absent from other metasearch engines. It can allow metasearching inside regions and return results that are entirely inside a specified country (or inside either one out of 2 countries). Opera-hostile?

[Dogpile]
http://www.dogpile.com/

Dogpile is a good, albeit akamai infested, metasearch tool to use if you are looking for a lot of information or something that is hard to find, need to use a more complex type of query, and don't mind getting some duplicate hits. Dogpile accesses many more search engines then any other metasearch tool. It does not rank or sort the results or eliminate duplicates. You get results from one engine after the other in the same format you would get if you visited the site and entered your query there. The results come in batches so you can deal with one group of results before going on to the next. Dogpile brings back the first 10 or 20 hits from each site. There is a button that allows you to bring back still more hits if you want them. Dogpile also lets you build your own customized search strategy. Dogpile and its companion Metafind are resources developed by the same person, and Metafind has now been fagocitated by Metacrawler.

There is also an interesting 'compare search engines results' tool, that will compare your (first 10 results) queryes on yahoo, MSN and gogle.

Search Options: You can select where to search first: The Web, Usenet, FTP.
And Then Options: Presents these choices: STOP (default), The Web, Usenet, FTP. This allows you to start with the targets on the Web list and go on with the Usenet or FTP targets. The default (sensibly) is to STOP.
Search Engines - searched three at a time in the group you have selected: The Web: Yahoo!, Lycos' A2Z, Excite Guide, World Wide Web Worm, WWW Yellow Pages, PlanetSearch, What U Seek, Lycos, WebCrawler, InfoSeek, OpenText, AltaVista, Excite & HotBot.; Usenet: Hotbot News, Reference.com, Dejanews, Infoseek News, Altavista and Dejanews' old Database.; FTP: Filez, FTP Search and Snoopie!.(Only the first word in your query will be passed on to these search engines.)
Speed: Relatively fast for each group of targets. Dogpile does not do any reformatting or sorting, so the results will be passed on as soon as they are received by Dogpile.
Query Options: Dogpile accepts Boolean queries and translate them for each search engine that is accessed. If you just type a list of words, the system treats it as if the words were joined by AND, equivalent to searching for all the words. The Boolean terms you can use are: AND, OR, NOT, NEAR. You can also use quotes to delimit phrases as in "search tips". Parentheses can be used to group terms. Some sites do not handle phrases, parentheses or some of the Boolean options. These will be deleted before the query is sent to the site. A moderately complex query: "search tips" hints NOT database is interpreted properly and translated appropriately for all the engines on the list.
Results: Results are presented without any reformatting just as received from each search engine. After the results from each engine, Dogpile adds a button allowing you to get more (if more are available).
Customized searching: The Custom Search button on the Dogpile home page takes you to a page where you can customize Dogpile by picking which search engines Dogpile will query and the order of the search. Your browser has to be capable of receiving "cookies" and cookies have to be enabled for you to use this option. Dogpile also provides an Advanced Search page that lets you select which engines will be searched but not the sequence of search.
Wait option: This option tells Dogpile how long to wait for each engine to return results. The default is 20 seconds.

[about, altavista, goto, infoseek, lycos, thunderstone, yahoo]

[MetaCrawler]
http://www.metacrawler.com/index_text.html

Metacrawler is the resource to use if you want quick, accurate results. It uses all of the most effective search engines except HotBot. The standard interface allows you to click fast or complete for quick and detailed searches respectively. The complete choice seems almost as quick as fast and returns a longer list of results. Metacrawler provides a [power user interface] that provides more control and search options, including an ability to filter hits by site location.

Search Engines: AltaVista, Excite, InfoSeek, Lycos, WebCrawler, & Yahoo!
Speed: Very fast. Results are typically returned in about 15 seconds, faster than searching many of the engines directly.
Query Options: You can select one of these options: any, all, as a phrase.
Results: Metacrawler combines the results from each search engine into a single list, eliminating duplicates and noting which search engine or engines detected each item in the list.
Power Search options
Results from: Everywhere, North America, Europe, Asia, South America, Africa...: Metafind filters the list of hits from each search engine, only including those with URL domains matching the region you have chosen. When using this option, it best to ask for 30 hits from each engine if you want to have something left after the hits have been filtered. Querying "how to search" with [Africa] produced no hits, with [North America] there were more than fifty.
Results per page: 10, 20, 30: Controls how many hits per page in the merged list.
Timeout: 5, 15, 30, 60... seconds.: This is how long it will wait for information from each site.
Results per site: 10, 20, 30: This is how many results it accepts from each search engine before eliminating duplicates and filtering on location.

[Ithaki]
[http://www.ithaki.net/dir.html]
"we're unable to detect your country, please choose it now" :-)

A rather weak server at 24.120.30.35, with "http://24.120.30.35/cgi-bin/alicia/nph-gogol.cgi?" if you choose Russia, "http://24.120.30.35/cgi-bin/alicia/nph-metabuscador.cgi?" if you choose Argentina, "http://24.120.30.35/cgi-bin/alicia/nph-bossa.cgi?" for Brasil and so on.

Metor
[Metor]
Not bad. A german Metaengine by Volker Carlguth: "A search and retrieval system that integrates information from hundreds of databases whose contents can not be reached by traditional search engines. Metor includes specialized databases, archives and catalogs..."

Queryserver
[Queryserver]
Quite Interesting time/access data, clusters reports

[Metager]
[http://meta.rrzn.uni-hannover.de/ (Uni Hannover)]
Has also a 'Teste Existenz' (test if page exist) function

"Bei den Metasuchmaschinen hat MetaGer die Nase vorn."
parallel search, AND OR booleans, eliminates doubles

[Surfwax]

Example query
http://www.surfwax.com/servlet/com.surfwax.FrontEnd.home?cmd=frames&search=advanced+searching&max=100&sort=relevance&x=25&y=10

Click the small lenses to get a quick summary (SiteSnap) Lenses with a red plus are "most relevant", red little stars represents Home pages, and then there is a list of the search engine used as "source" for each given result... (hotbot, yahoo, open directory...)

[Ez2find] (ex ew2www)

Ez2Find is a good French metasearch engine that gathers results from various main search engines, parses the results, removes the duplicates and includes links to relevant directory categories (results from the Open Directory) and to clustered results.

http://ez2find.com/meta/global/search.mpl?mode=all&per_page=20&timeout=10&depth=1&safe=&qry_str=fravia&category=Any+Language

Note the "clustered results" on ez2find's right side!

Ez2find also offers Systran's translation (pseudo-proxi) service.

[Iboogie] ~ [Advanced Iboogie]
A typical query: http://iboogie.tv/searchtree.asp?name_query=advanced+searching&name_tab=0&name_do_search=1&name_news_tab=0
(&name_sources=MSN%3BAllTheWeb%3BWiseNut%3BNetscape%3BOverture%3B&name_sources_current=MSN%3BAllTheWeb%3BWiseNut%3BNetscape%3BOverture%3B&name_tab_current=0)
Image query:
http://iboogie.tv/searchtree.asp?name_query=steam+gwr&name_tab=2&name_do_search=1&name_news_tab=0&name_sources=picsearch%3Balltheweb_img%3B&name_sources_current=picsearch%3Balltheweb_img%3B&name_tab_current=2
(compare with Fast's image search)

Metasearch for images as well
"The algorithms developed by iBoogie are using a combination of linguistic clustering and statistical clustering. They generate hierarchical clustering as opposed to a simple "flat" grouping of similar documents. This is done in real-time on a set of documents return by the search, without any predefine grouping, pre-build knowledge base, or pre-processing of all the document collections used by the search engines."

"Iboogie's clustering is computationally inexpensive and very fast, it will process 250 text snippets in about 140 milliseconds on a Pentium III, 864Mhz with 256MB of main memory"

[search.com] ~ [Advanced search.com]
A typical query: http://www.search.com/search?tag=se.fd.box.main.search&q=advanced+search

"Results from: Google, AltaVista, Ask.com, Business.com, Kanoodle, LookSmart, MSN, Miva, and more"

[Ilectric]

Nice meta with images and links together.
"With a single query, you get the most comprehensive results from Altavista, Teoma, Alltheweb, Amazon, Sprinks, DMOZ, Yahoo, and Kanoodle. For every query, ilectric metasearch sorts and ranks each hit, removes duplicates, and presents the end result with inline images"

Corrections and Additions

Dave's (March 2000)

Gheez Fravia+: I found an interesting and very helpful search technique encapsulated at www.quickbrowse.com that I didn't see referenced by URL or concept on your new pages. Concept is simple... goes to desired engine and downloads ALL the pages of hits from your search. No more "For the next 20 hits, click here and wait 20 seconds" stuff. You just get a long HTML doc that has all the hits from that engine. There is a onetime login process that take about 30 seconds and a cookie no doubt, but after that you can use the "Quicksearch" link and the rest of the iterations of the quickbrowse implementation.

Makes life a lot easier for me, and thought you might want to include it in your excellent pages of lores. I have learned SO much from you over the years, thanks for sharing your knowledge!

Dave Lamme (California)

Jeremy's (December 2001)

Sir, on your webpage relating to compound search I dont see the search site which I use: Queryserver
http://www.queryserver.com/web.htm
It uses 10 search engines and returns categorised results. I have no connection with this site other than as a (fairly) satisfied user.
(I was studying your site in the hope of finding something better :*)
_______________________
Jeremy L Hinton

Wanna search Metacrawler?

Note that Metacrawler has now fagocitated Metasearch, yet another nice example of oligarchical development and diminishing variety, typical of the impoverished webworld of the 'commercial bastards'.


	Have a go!
	any all phrase	[metacrawler.com/help/faq/]

"Metameta" search

Yup, a Metameta search. A tool for searchers that want to cast a big net over the pond and want to catch only the biggest and most visible fishes. Of course this kind of search multiplies ad libitum the known disadvantages of all metasearch engines as well: spending a very short time in each, already "shorttimed", meta-data-base...
Note also that the metameta bot below pumps from some metaengines (which are not listed above) that I would not actually recommend

CleverSearch meta search engine
Enter words to search for in the box below:

Choose index:

Results per page:

Match:

Engines to use:

MetaCrawler
ProFusion
HuskySearch
SavvySearch

MetaFind
Mamma
Dogpile
Surfy

Compound pointers

A big thank to friend Iefaf

As a first try: www.metaeureka.com

More links:
Metaværktøj (metasearch in Danish)
Lists 25 Meta-search Engines (but no super-meta)
http://www.db.dk/dbi/internet/metavrkt.htm

A guide to specialized SE
http://www.searchability.com

A huge list
http://www.leidenuniv.nl/ub/biv/specials.htm

Another one (340K)
http://natur.exit.mytoday.de/koi/Special_Links_Search.htm

iRover was nice but now dead.
http://irover.linuxave.net/cgi-bin/isearch?

http://it.umary.edu/Library/research/support/search_engines.html

Vivisimo (sic): the correct spelling would be vivissimo :-)
See the essay by Shally Steckerl, that points out how Vivisimo "goes miles above and beyond simply getting content from other search services"!

A typical Vivisimo's query: http://vivisimo.com/search?query=%22advanced+searching%22&v%3Asources=Web&x=35&y=16

Fazzle

Quite good meta-search engine. Allows boolean searches!
A typical Fazzle's query: http://www.fazzle.com/search?SearchType=136&SearchString=advanced+searching&Filter=4&Filter=2&Language=0
Fazzle's Advanced search


	And		Or		Phrase		Title		Url		Boolean

Web Images Audio Video News Yellow Pages White Pages
		Preferences Advanced Web Search

Query queryserver for:	Help!
		Search
	Customize