Using Best Bets to Guide Search Results

Each week you receive a report detailing the search queries of your visitors. Have you taken a look at the most popular queries historically? The queries listed here are good candidates to add to a Best Bets list.

Best Bets allow you to hand-pick pages on your website to appear at the top of the search results for specific queries. Here is how it works:

1. You provide a few must-match terms.
2. You also provide a URL and a short description.
3. When a search includes all of the must-match terms, the URL and description are listed at the top of the search results.

To specify Best Bets, log in to Blossom.com and follow the link Best Bets in the Search Index Settings section:

• The Title will be the heading above the Best Bets. The default is "Best Bets".
• To add a Best Bet, select Add a New Item.
• If a query contains all of the Terms specified, then the Best Bet will be shown. Terms may include the wild card characters "?", to match any one character, and "*" to match zero or more characters.

Redirection program parameters encrypted

Search engines, being complex programs, are a frequent target of hackers looking to exploit a server. A common exploit is to get the search engine to perform an unintended task, such as running an arbitrary system command or getting the server to act as a proxy.

If you use the Blossom adaptive search engine or page highlighting, then you may have noticed a change that prevents a hacker from using the search engine as a proxy. Both adaptation and page highlighting wrap URLs in the search results list with a call to a redirection program. The redirection program retrieves the document at the URL and then does additional processing on the document.

When it was first introduced, the URL for the page to be retrieved was a clear parameter to the redirection program. That attracted hackers wishing to send traffic to websites that appeared to come from Blossom. We will leave it to your ingenuity to figure out ways that might be exploited, but we noticed thousands of redirects daily not coming from search results. Hackers had found an exploit.

To solve the program we now encode the URL to be retrieved. You will see the encoding if you look at the link for the items in a search results list. It is not as informative as having the URL be readable, but it is a lot safer.

By the way, keeping hackers off your server is one nice benefit of using a search service. Besides eliminating the traffic from hacker's probes, should a hacker find an exploit they would still be isolated from your server. It becomes a problem for us, but not for you.

Adaptive ranking of pages in search results

The next revision of the search engine will include the facility to have prior searches influence the order in which documents are listed in the search results. Over time, those pages selected most often by searchers will be rise in the list. To accomplish this, the search engine will keep track of what people select from the search results list; that is, the links in the search results will go indirect through a Blossom search server. Yahoo! search has done this for many years, and Google started doing it within the past couple of years.

Sound interesting? To enable adaptation to your index, just append /adapt to the search URL in your search forms. The links in the search results will then point back to a search server. To prevent hacking, the URLs are encoded.

Blossom indexer now understands Open Graph document dates

The search engine factors in the age of a document to determine its position in the search results list. Naive treatment of dynamically generated documents, e.g., pages from ASP or PHP scripts, degrades the quality of search results, because web servers don't report their age accurately. The heart of the problem is that the web server doesn't know the real last-modified date of a dynamically-generated document, so it tells the Blossom spider the document was last-modified "today".

The solution is to ignore the web server and encode the real last-modified date within the document itself. For many years, the Blossom indexer has used the meta tag http-equiv="Last-Modified"  to override the date reported by the web server (see the Search Guide FAQ for details).

With the advent of standards for semantic mark-up of web pages, we are adding other encodings for the last-modified date. Currently we are testing the Open Graph protocol. In particular, the indexer will look in the HTML head section for

  <meta property="article:published_time" content="YYYY-MM-DD">
or
   <meta property="article:modified_time" content="YYYY-MM-DD">

We will add other protocols to the indexer as the need arises. Let us know if you are using a different protocol by sending email to Blossom Support.

Reporting tools for viewing an index reorganized and enhanced

We have reorganized and expanded the tools available for viewing a search index. All of the tools are now accessible from the Retrieve details about the last update of an index link in the Actions section of the Account Settings page. These tools are available:

• Index summary: page count, date of last update, and list of start URLs
• List of URLs for the documents in the index
• The meta data for each URL, including the date (as interpreted by the search engine), the page title, description, keywords, and key phrases.
• The link structure of the index. For each document in the index, it lists the documents it points to and the other documents pointing to it.
• The broken links encountered during spidering.
• The PDF files ignored. Sometimes PDF files are used as a container for scanned documents. The scanned documents are pictures to the search index and so are not scanned. To save download bandwidth, these image PDF files are not reread.

Blossom has moved to the cloud

For the past month, Blossom has been testing virtual servers from Amazon Web Services. Our experience has been very positive and so we have moved all our spiders and search servers to the cloud. Virtual servers provide several benefits, including excellent reliability and easy scaling.

Since its inception, Blossom has used redundant servers to achieve high reliability. For the past ten years that has worked well as Blossom Search has never been completely offline. When a server went down, others would pick up the search traffic. To minimize hardware failure, we replaced server hardware on average every two years.

Using virtual servers eliminates our need to replace hardware as the underlying infrastructure is maintained by the service provider, in this case Amazon. In addition, because the capability of the hardware is specified in software, we can expand server capacity quickly without interruption of service. 

If you monitor Blossom's spider, you will now find that all requests appear to come from IP address 54.201.166.110. Our search servers are now all also behind a single gateway at 54.201.121.129. With the new configuration, mirroring is now continuous, so you won't find any update delays as in the past and there is no longer any need to test on "search1.blossom.com". Your changes should be visible in just a few seconds on all servers.

Movement to the cloud has been a major change for us. Although we have spent many hours testing, there is always a chance we've overlooked something. Please send us email if you find anything amiss.

Improvements to indexing and searching

Over the past few months we have been working through our "would be nice" list of improvements. Among the recent changes to the indexer and search engine are:

• Better support for common HTML mistakes, such as unterminated tags and misformated date meta tags.
• Improved date sorting. Previously, documents older than a few years were grouped together.
• Expanded support for structured data, including tel: and mailto: protocols.
• Further refinement of the weighting function to generate more relevant result lists.

We have also been enhancing the tools for visualizing the contents of an index. To see the tools available, log in to the Blossom site and look for the Actions section at the bottom of the Settings page.