Improvements to indexing and searching

Over the past few months we have been working through our "would be nice" list of improvements. Among the recent changes to the indexer and search engine are:

• Better support for common HTML mistakes, such as unterminated tags and misformated date meta tags.
• Improved date sorting. Previously, documents older than a few years were grouped together.
• Expanded support for structured data, including tel: and mailto: protocols.
• Further refinement of the weighting function to generate more relevant result lists.

We have also been enhancing the tools for visualizing the contents of an index. To see the tools available, log in to the Blossom site and look for the Actions section at the bottom of the Settings page.

Expanded tools to analyze search indexes

We have been expanding and improving the tools you can use to understand a search index. This note discusses three of these tools, accessible from the "Actions" section of your Search Configuration page by following the link Retrieve the list of URLs, meta data, and hyperlinks in an index.

1. The list of URLs is extracted directly from a search index. It tells you the exact source documents that were used to generate the index.
2. Meta data includes such items as the title for a page and the date the page was last modified. The meta data is shown for each of the URLs in the index; that is, it is the same URL list shown by the "List of URLs" tool.
3. The list of hyperlinks from each page comes from the spidering log and it includes links both on and offsite. It tells you what links are on a page and what pages contain a particular link. It can answer the question "Why is this page in the index?".

You may notice more URLs in the "List of Hyperlinks" than in the "List of URLs". During indexing we carry out a more complete duplicate removal than that implemented during spidering. As a result, the spidered list may include pages that are removed during indexing.

You may also notice that the Starting URLs in the "List of Hyperlinks  varies depending on whether the last spidering was complete or incremental. On a complete respidering, the starting URLs are just those specified in the "Include List" for the index. For an incremental spidering, the starting URLs are all the URLs in the current index.

Modified scoring function

We have been revising the search engine's scoring function that determines the order pages are listed when they match a search query. PDF files, in particular, are troublesome because they often are archival documents and not a good match for the search queries of typical site visitors. For that reason, PDF files are penalized in their score relative to an HTML document with the same content. The penalty function has recently been rewritten. In general, PDF files are usually scored lower in the new function, but now always. Your feedback on the search results is always welcome.

If you are not happy with the default scoring, you can influence the search engine's scoring of pages by using the "/pdf" option, adding keywords and keyphrases to pages, and by setting page weights explicitly. Each of these techniques is described briefly in the Search Guide, and in more detail in the library document Fine tuning with page weights and meta tags.

Block link style now the default

Link style 3, described in our previous post, is now available in the standard search engine--in fact it is now the default style. If you do not specify a style in your search URL, the block style is used.

To get a different style, add "/linkN" to your search URL, where N is the style number. The previous default style was 0 (zero), so to get style 0 add "/link0". For a complete description of the link styles, please see Output Format section of the Search Guide.

New block link style in search results

We are testing a new link style that turns the extract for a page into a block. The block has a few features that enhance the search results:

• As a user moves their pointer over the block, the block is highlighted.
• While the pointer hovers over the block, the block expands to show more sentences containing the search terms (if any).
• Clicking anywhere in the block brings up the matched document.

The new link style is #3, thus to use it add "/link3" to your search URL. During testing it will be available only in our test search engine, nquery. (That is, use nquery in place of query in your search URL.) You can try it out by searching this blog!

New scoring function for PDF files

The scoring function for PDF files has been changed to achieve two goals:

1. By default, PDF files are penalized so that they appear lower in search results than other document types, such as HTML. This has always been the case, but the penalty has been increased. Still, highly relevant PDFs may appear before less relevant HTML.
2. The /pdf0 option now imposes a much greater penalty. Appending "/pdf0" to a search URL adds a significant penalty to PDFs. In general this means that in the search results PDFs will follow other file types, such as HTML. However, as in the default case, very relevant PDFs may rise in the search output.

Enhanced semantic entities

We have completed a significant upgrade of processing for semantic entities. Semantic entities are items like phone numbers and prices. The basic facility allows queries to match patterns of text that represent instances of the entity. For example, the query "product price", besides matching sentences containing the words "product" and "price", also matches sentences containing "product" and an instance of a price such as "$25". Currently supported entities are: phone numbers, email addresses, postal codes, prices, and times of day.

Are there entities you would like to see added? Send suggestions to Blossom Support.