search engines: Encouraging and Discouraging

Encouraging and Discouraging Search Engines from Indexing your Web Page

The (anti-) Robots META tag

The robots meta tag is placed in the HTML code of a web page between the <HEAD></HEAD> tags. It tells visiting robots/spiders not to index that particular web page and not to follow any links the web page may have.

Note that not all search engines will recogize the robots meta tag.

Example:
<HEAD>
<TITLE>
My Secret Life
</TITLE>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</HEAD>

Full details on how this tag works is provided at:

Web Server Administrator's Guide to the Robots META tag
HTML Author's Guide to the Robots META tag

Information from http://info.webcrawler.com/mak/projects/robots/exclusion.html#meta

Another method is to create a list of all the pages on a web site that are to be shielded from spiders and save it to the root of the server as a file called "robots.txt." This is generally done by the web site's webmaster and not an option for individual web authors.
For more information, see http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html

Methods for Encouraging Indexing

What Spiders Look For:

"All search engines rank their results so that more relevant documents are at the top of the list. This sorting isbased on, first, the frequency of keywords within a document, and second, the distance of keyword occurrences from the beginning of the document.

           "In other words, if one document contains two matches for a keyword and another is identical but contains
           only one, the first document will be closer to the top of list. If two documents are identical except that one
           has a keyword positioned closer to the top (especially, in the document title), it will come first.

           "In addition to these principles, some search engines use extra factors to determine the ranking order, called
           relevancy boosters. For instance, HotBot and Infoseek favor those documents that make use of META tags
           over their METAless peers.

           "WebCrawler relies on link popularity: if a page is linked frequently from other pages and sites, it is
           considered "more authoritative" and gets some priority on the list of results. Excite, being a combination of a
           search engine and a directory, quite naturally gives preference to those pages that are reviewed in its
           directory.

           "Finally, all search engines try to fight unfair practices of some webmasters who attempt to fool the ranking
           algorithm by repeating keywords to improve their effective frequency in the documents. You might have
           noticed pages with a tail of hundreds of repeated keywords (usually made invisible in browsers by changing
           font color, but still visible to search engines) or pages with multiple TITLE elements (again, only the first one
           is visible in browsers, but all are indexed by a spider). Now, not only do such "keyword spammers" not
           receive high rankings, but many search engines also automatically exclude them from the database. (For
           more on spamming, see "The Meta Controversy" later in this chapter.)"

Source: HTML Unleashed PRE. Strategies for Indexing and Search Engines: Search Interface - webreference.com

Adding Meta Tags

Meta tags are notes placed in the HTML source of a web page between the <head></head> tags. There are two main types that draw the attention of search engines and help them catalog what they find: Keywords and Description. The former simply lists a string of search-type keywords the author feels apply to the contents of his page. The latter is generally a one-sentence statement about the nature of the page. Description meta tags are used by many search engines as the brief description you see when viewing search results.

The following is an example of a web page head section using both Keyword and Description meta tags.

A free online service called the Meta-tag Generator can help you construct your meta tags. You input the information, they email the tags to you, and you simply copy and paste the tags into your web page code.

Registering Your Site

Metatags will tell a search engine's robot what to make of your page when it comes across it but it also pays to actually register your site directly with the engines. With services such as Yahoo! the only way to get listed is to go to www.yahoo.com and click on the "suggest a site" link. Others rely on what their robots bring to them.After you have added descriptive metatags to your page, there is a free service that will submit the URL for your web page to most of the major search engines in one step and reportedly draw their robots to you to read your tags. Go to http://www.fg-a.com/RegisterYourSite.htm

Page created by Dann Foster, dannf@oakton.edu
Last update 3/17/00