Targeted incremental updates for fast turnaround

There are now three ways to update a search index:

1. Automatically. Periodically our spider visits your website and looks for new pages. The process is fully automatic. Normally you should rely on this automatic update.
   
2. Manually. From the Search Configuration page at Blossom.com, you can trigger a complete rebuild of your index. This makes sense after a major reworking of a site. To trigger a rebuild, follow the "Update an index" link and click on "Full Rebuild".
3. Incrementally. When you add new pages to your site and you'd like them to appear quickly in your index, you can perform an incremental update where you tell us where the new pages are. To trigger an incremental update, follow the "Update an index" link and click on "Incremental Update". You'll be asked for a file containing the URLs of the new pages.
   

Customizable entities on the way

A significant upgrade of the Blossom search system is in development. One new feature is a facility for defining custom entities. The search engine has several entity types built in, such as phone number, email addresses, and time of day. An entity consists of two parts: a trigger and a pattern. When a trigger is detected in a query, it can be replaced by text that matches the pattern. For example, a search for "office phone number" would match not only the words "office phone number" but also the word "office" near a phone number, like "office: 888-123-4567". The custom entity facility will allow site designers to define their own triggers and patterns. More details will be provided when the facility is available for testing. In the meantime, to help us design the facility, we are interested in hearing about how you might use custom entities. If you think you have a use for them, please send a note support@blossoft.com.

List of image-only PDF files

Some sites have PDF files that don't contain text; often they are generated from scanned documents. The lack of text makes the files invisible to the search engine. To reduce the number of times image-only PDF files are downloaded, a list is kept for each index. You can now download that list from the Search Configuration page. Follow the link "Retrieve the list of image-only PDF files being ignored" in the "Search Index Settings" section.

The list is cleared whenever a complete respidering of the index is triggered, such as when you request a non-scheduled index update (by following the "Update an index" link on the Search Configuration page).

New option for database-driven sites

Some websites have pages generated dynamically from a repository. Often on these sites there are multiple URLs that can generate the same page. Blogs are an example; an article may have multiple labels, each label providing a path to the same text. For example, www.domain.com/july/article1 and www.domain.com/announcements/article1 might both refer to the same article.

A new option has been added to the Blossom spider telling it to ignore all but the last component of a URL when determining whether two URLs are the same. Thus in the example above, "article1" would only be retrieved once. This will reduce the number of duplicate documents downloaded from a site, saving both bandwidth and potential page count.

Contact Blossom Support if you think your site might benefit from using this option.

Spaces in URLs

Over the past few weeks we've been experimenting with different rules for handling space characters in URLs. Technically, spaces are not allowed in URLs; they should be encoded as %20. Nevertheless, many websites use spaces in file names.

We looked at treatment of spaces in common browsers. In every case browsers translate interior spaces to %20. The treatment of leading and trailing spaces varies. The most forgiving behavior is to remove them. Thus we now remove leading/trailing spaces, but keep interior spaces. Here are a couple of examples to illustrate:

<a href=" leadingSpace.html"> is treated as <a href="leadingSpace.html">

While

<a href="interior Space.html"> becomes <a href="interior%20Space.html">

Page highlighting for Word documents

Page highlighting is the feature that highlights a searcher's search terms on a retrieved page. Up until now, page highlighting was available only for HTML and PDF files. Now page highlighting works for DOC files as well. To use page highlighting, add the "/get" option to your search command. Details on usage are available in the Search Guide under Output Format.

New handling of file types

File-type filtering allows searchers to limit search results to documents of a particular type, such as HTML or PDF files. Previously, file-type filtering needed to be enabled explicitly in the search URL using the "type" option. In the latest search engine, the feature is on by default. To turn it off, use the "notype" option.

Also previously, file-type filtering was disabled when categories were used. This restriction has been removed, so search results can now be filtered both by file type and by topic category.