Invisible web scanner

SOME ASTRONOMERS hold the belief that galaxies are moving away from one another and the universe is expanding. Likewise, the web universe, with thousands of sites that get added everyday, is expanding rapidly. Though this should be happy news for Netizens, they should not fail to see the fact that most of these pages are invisible to the conventional search engines - Netizen's main `spectacle' to scan the Net.

Invisible web

When we look for sites that contain the required service/information, we usually access a search engine and invoke a search process; the search engine scans its database, collects all the appropriate links that match our query, cooks a web page with this search result and pushes it to our browser - the richness/validity/newness of the result depends on the content/quality of the engine's database.

One of the major methods through which a search engine expands its web database is by grabbing the link information using automatic programs called crawlers/spiders. These programs move continuously from one site to the other and fetch information and push them into the engine's database. This technique works fine as long as the scanned site has a well-laid out site structure with static web pages. But the web technology has gone quite far and these days static web sites are almost on the way out. Of late most sites get created on the fly, from a database that houses the various web page components. Depending on the query and the user, every time the service dishes out a new customised page.

As the web content lies in the database, conventional search engines are unable to locate this using the normal methods. This part of the web, which is not visible to the search engines, is known as invisible web or deep web. It is estimated (from a Bright Planet study) that this part of the web is many hundred times bigger than the part of the web that can be seen by the search engines. If you want to learn more about this intriguing aspect of the Net, you may visit the following sites:

http://www.completeplanet.com/tutorials/deepweb/index.asp; and

http://www.freepint.co.uk/issues/080600.htm#feature

LexiBot

LexiBot excellent search tool, developed by Bright Planet (http://brightplanet.com/), can be effectively used to meet the challenges posed by the invisible web phenomenon. The software that has the necessary features to submit queries directly to databases brings us the treasures of information that lie in the databases. Apart from this ability to send directed queries during a search process, the utility has the capability to invoke multiple search engines simultaneously to accomplish the search job. The program also provides the option to select the web sources or search engines to be employed for this search process from a list of around sixty web sources.

The software does not finish its job by just downloading the links; it analyses the links, removes irrelevant links, then downloads the pages (without pictures) and stores it in your hard disk. The service also provides the facility to save the search results in your machine so that you can retrieve it later. So, no more the dead links problem; no need to worry about storing results; no need to even invoke your browser as the utility has its own text viewer. LexiBot does all these activities in the background allowing you to do other things - what a responsible and resourceful tool! The program has many more features, which I feel you can try out yourself. This free software (2.3 MB) can be downloaded from the site: http://www.lexibot.com"

Another site, which has the capability to unearth the web resources that lie in the invisible web, is http://www.invisibleweb.com

Update: E-mail encryption

It is a well-known fact that your e-mail messages can be intercepted and read by anybody who wants to do it. This column addressed the issue of e-mail security a year ago (refer The Hindu dated February 23, 2000). One of the most effective techniques that can be used to curb this menace is e-mail encryption - encryption is the process of altering the message with the help of some mathematical algorithm so that it can be read by the recipient only if he has the necessary information to decrypt it. Though there are many techniques available that let you send/receive encrypted messages, the proportion of the e- mailing community that makes use of this feature is rather negligible.

One reason for this slow adoption of e-mail security measures - in spite of the almost paranoid propaganda on the insecurity factor in cyber world communication system - can be attributed to the technical skills demanded by most e-mail security schemes. Most e-mail users are non-technical and do not have the necessary skills or inclination to learn about and install an encryption package. What they want is a simple solution that encrypts and decrypts the messages without any effort from their end.

Do you fall in this category? If yes, here is a neat and simple solution that lets you send encrypted e-mails effortlessly. The program, Encryption Plus, allows you to optionally encrypt the mails you send and the solution does not demand the recipient to have the program to decrypt the encrypted mail. When you send the mail in an encrypted form, you need to provide a pass phrase; the program encrypts the mail content using this pass phrase and sends the mail as an attached executable program file. The mail will reach your friend's box as an attachment file.

To read the message, which has come as an executable file (a file with extension .exe), the recipient has to just double-click on the file's icon and has to provide the pass phrase when prompted for it. Once the recipient gives the proper pass phrase - used to encrypt the mail by the sender - the message content gets loaded in an ordinary text viewer, which can be stored as a text file.

One drawback of the program is that among the popular mail clients, the software supports only Outlook. To install the program and integrate with Outlook, download the Encryption Plus Email utility (1.3 MB) from the site: http://www.pcguardian.com/downloads/index.html and run it. After successful installation, invoke Outlook and integrate it with the Encryption Plus. (Click at tools, the options, access the `Encryption Plus for Email' button and enable the appropriate boxes).

J.Murali

(The author can be contacted at: murali27@satyam.net.in)

Send this article to Friends by E-Mail

Section  : Science & Tech
Next     : Evidence of slushy Ganymede found