List of image-only PDF files

Some sites have PDF files that don't contain text; often they are generated from scanned documents. The lack of text makes the files invisible to the search engine. To reduce the number of times image-only PDF files are downloaded, a list is kept for each index. You can now download that list from the Search Configuration page. Follow the link "Retrieve the list of image-only PDF files being ignored" in the "Search Index Settings" section.

The list is cleared whenever a complete respidering of the index is triggered, such as when you request a non-scheduled index update (by following the "Update an index" link on the Search Configuration page).

New option for database-driven sites

Some websites have pages generated dynamically from a repository. Often on these sites there are multiple URLs that can generate the same page. Blogs are an example; an article may have multiple labels, each label providing a path to the same text. For example, www.domain.com/july/article1 and www.domain.com/announcements/article1 might both refer to the same article.

A new option has been added to the Blossom spider telling it to ignore all but the last component of a URL when determining whether two URLs are the same. Thus in the example above, "article1" would only be retrieved once. This will reduce the number of duplicate documents downloaded from a site, saving both bandwidth and potential page count.

Contact Blossom Support if you think your site might benefit from using this option.

Spaces in URLs

Over the past few weeks we've been experimenting with different rules for handling space characters in URLs. Technically, spaces are not allowed in URLs; they should be encoded as %20. Nevertheless, many websites use spaces in file names.

We looked at treatment of spaces in common browsers. In every case browsers translate interior spaces to %20. The treatment of leading and trailing spaces varies. The most forgiving behavior is to remove them. Thus we now remove leading/trailing spaces, but keep interior spaces. Here are a couple of examples to illustrate:

<a href=" leadingSpace.html"> is treated as <a href="leadingSpace.html">

While

<a href="interior Space.html"> becomes <a href="interior%20Space.html">

Page highlighting for Word documents

Page highlighting is the feature that highlights a searcher's search terms on a retrieved page. Up until now, page highlighting was available only for HTML and PDF files. Now page highlighting works for DOC files as well. To use page highlighting, add the "/get" option to your search command. Details on usage are available in the Search Guide under Output Format.

New handling of file types

File-type filtering allows searchers to limit search results to documents of a particular type, such as HTML or PDF files. Previously, file-type filtering needed to be enabled explicitly in the search URL using the "type" option. In the latest search engine, the feature is on by default. To turn it off, use the "notype" option.

Also previously, file-type filtering was disabled when categories were used. This restriction has been removed, so search results can now be filtered both by file type and by topic category.

Improvements in Blog Search

One of the biggest challenges in indexing a blog is to get the correct date for a posting. Blog software generates web pages dynamically, so Web servers usually report a "last modified" date of today. We continue to expand the number of blogging systems for which the Blossom indexer can identify the posting date separately from the Web server reported date. Please let us know if your search index includes a blog and the posting date is not handled correctly.

We have also been working to improve spidering of blogs by identifying archival posts from currently active posts. Archival posts are now spidered less frequently, allowing us to reduce significantly the load spidering places on a blog.

Best Bets now available with Affinity Search

The Best Bets feature lets you guarantee that specific pages will appear at the top of the search results when your visitors search for particular keywords. For example, when someone enters the term "emergency", you might want a page of emergency numbers to appear first in the search results. Until now, Best Bets were only available with Enterprise Search, but now it has been added to Affinity Search as well. For details on using Best Bets, please see the Search Guide section Search Options.