Adaptive ranking of pages in search results

The next revision of the search engine will include the facility to have prior searches influence the order in which documents are listed in the search results. Over time, those pages selected most often by searchers will be rise in the list. To accomplish this, the search engine will keep track of what people select from the search results list; that is, the links in the search results will go indirect through a Blossom search server. Yahoo! search has done this for many years, and Google started doing it within the past couple of years.

Sound interesting? To enable adaptation to your index, just append /adapt to the search URL in your search forms. The links in the search results will then point back to a search server. To prevent hacking, the URLs are encoded.

Blossom indexer now understands Open Graph document dates

The search engine factors in the age of a document to determine its position in the search results list. Naive treatment of dynamically generated documents, e.g., pages from ASP or PHP scripts, degrades the quality of search results, because web servers don't report their age accurately. The heart of the problem is that the web server doesn't know the real last-modified date of a dynamically-generated document, so it tells the Blossom spider the document was last-modified "today".

The solution is to ignore the web server and encode the real last-modified date within the document itself. For many years, the Blossom indexer has used the meta tag http-equiv="Last-Modified"  to override the date reported by the web server (see the Search Guide FAQ for details).

With the advent of standards for semantic mark-up of web pages, we are adding other encodings for the last-modified date. Currently we are testing the Open Graph protocol. In particular, the indexer will look in the HTML head section for

  <meta property="article:published_time" content="YYYY-MM-DD">
or
   <meta property="article:modified_time" content="YYYY-MM-DD">

We will add other protocols to the indexer as the need arises. Let us know if you are using a different protocol by sending email to Blossom Support.

Reporting tools for viewing an index reorganized and enhanced

We have reorganized and expanded the tools available for viewing a search index. All of the tools are now accessible from the Retrieve details about the last update of an index link in the Actions section of the Account Settings page. These tools are available:

• Index summary: page count, date of last update, and list of start URLs
• List of URLs for the documents in the index
• The meta data for each URL, including the date (as interpreted by the search engine), the page title, description, keywords, and key phrases.
• The link structure of the index. For each document in the index, it lists the documents it points to and the other documents pointing to it.
• The broken links encountered during spidering.
• The PDF files ignored. Sometimes PDF files are used as a container for scanned documents. The scanned documents are pictures to the search index and so are not scanned. To save download bandwidth, these image PDF files are not reread.

Blossom has moved to the cloud

For the past month, Blossom has been testing virtual servers from Amazon Web Services. Our experience has been very positive and so we have moved all our spiders and search servers to the cloud. Virtual servers provide several benefits, including excellent reliability and easy scaling.

Since its inception, Blossom has used redundant servers to achieve high reliability. For the past ten years that has worked well as Blossom Search has never been completely offline. When a server went down, others would pick up the search traffic. To minimize hardware failure, we replaced server hardware on average every two years.

Using virtual servers eliminates our need to replace hardware as the underlying infrastructure is maintained by the service provider, in this case Amazon. In addition, because the capability of the hardware is specified in software, we can expand server capacity quickly without interruption of service. 

If you monitor Blossom's spider, you will now find that all requests appear to come from IP address 54.201.166.110. Our search servers are now all also behind a single gateway at 54.201.121.129. With the new configuration, mirroring is now continuous, so you won't find any update delays as in the past and there is no longer any need to test on "search1.blossom.com". Your changes should be visible in just a few seconds on all servers.

Movement to the cloud has been a major change for us. Although we have spent many hours testing, there is always a chance we've overlooked something. Please send us email if you find anything amiss.

Improvements to indexing and searching

Over the past few months we have been working through our "would be nice" list of improvements. Among the recent changes to the indexer and search engine are:

• Better support for common HTML mistakes, such as unterminated tags and misformated date meta tags.
• Improved date sorting. Previously, documents older than a few years were grouped together.
• Expanded support for structured data, including tel: and mailto: protocols.
• Further refinement of the weighting function to generate more relevant result lists.

We have also been enhancing the tools for visualizing the contents of an index. To see the tools available, log in to the Blossom site and look for the Actions section at the bottom of the Settings page.

Expanded tools to analyze search indexes

We have been expanding and improving the tools you can use to understand a search index. This note discusses three of these tools, accessible from the "Actions" section of your Search Configuration page by following the link Retrieve the list of URLs, meta data, and hyperlinks in an index.

1. The list of URLs is extracted directly from a search index. It tells you the exact source documents that were used to generate the index.
2. Meta data includes such items as the title for a page and the date the page was last modified. The meta data is shown for each of the URLs in the index; that is, it is the same URL list shown by the "List of URLs" tool.
3. The list of hyperlinks from each page comes from the spidering log and it includes links both on and offsite. It tells you what links are on a page and what pages contain a particular link. It can answer the question "Why is this page in the index?".

You may notice more URLs in the "List of Hyperlinks" than in the "List of URLs". During indexing we carry out a more complete duplicate removal than that implemented during spidering. As a result, the spidered list may include pages that are removed during indexing.

You may also notice that the Starting URLs in the "List of Hyperlinks  varies depending on whether the last spidering was complete or incremental. On a complete respidering, the starting URLs are just those specified in the "Include List" for the index. For an incremental spidering, the starting URLs are all the URLs in the current index.

Modified scoring function

We have been revising the search engine's scoring function that determines the order pages are listed when they match a search query. PDF files, in particular, are troublesome because they often are archival documents and not a good match for the search queries of typical site visitors. For that reason, PDF files are penalized in their score relative to an HTML document with the same content. The penalty function has recently been rewritten. In general, PDF files are usually scored lower in the new function, but now always. Your feedback on the search results is always welcome.

If you are not happy with the default scoring, you can influence the search engine's scoring of pages by using the "/pdf" option, adding keywords and keyphrases to pages, and by setting page weights explicitly. Each of these techniques is described briefly in the Search Guide, and in more detail in the library document Fine tuning with page weights and meta tags.