New indexer live

A new spidering and indexing engine for Blossom Search is now live with significant changes to improve site coverage and search results. Over time, websites have become much more dynamic with much of the HTML generated at the time of delivery. Dynamic sites present two significant problems:

1. Links to some content may only be generated by client-side programs.
2. Multiple links may generate the same content.

These are not, of course, new concerns, but the increasing complexity of websites makes the spidering and indexing tasks more difficult. The new changes address both of these issues. As a result, you may see your search indexes grow, or perhaps shrink!

The index will grow if your site uses sitemaps. The Blossom spider will now routinely look for sitemap.xml in the root directory for a website. If the file exists, it will use the sitemap to help guide the traversal of a website. If you wish to prevent that behavior, log into the Search Configuration page for your index at Blossom.com, follow the "Spidering, Indexing, and Reporting" link and uncheck the box "Read Sitemaps".

The index will shrink if your site contains or generates very similar pages accessible by different URLs. Duplicate recognition has been improved to pick up pages with nearly identical content regardless of the tag structure. Duplicate pages always have the same title, so using unique titles for unique content will prevent the indexer from ever judging very similar pages to be duplicates.

Use sitemaps to guide Blossom spider

As websites become more dynamic, some links may be generated programmatically rather than specified directly in HTML. While the Blossom spider does search Javascript for URLs embedded in strings, it does not execute Javascript. As a result, URLs generated by string operations can be overlooked.

The spider was recently enhanced to read sitemaps as a way to guide its traversal of a site. By specifying a sitemap in the "include" list for an index, the spider will visit each URL in the sitemap.
 A sitemap is an XML file that lists the URLs on a website. (See https://www.sitemaps.org/ for details.)

For Blossom, the list doesn't have to contain all the URLs on a site; it only needs to include those URLs generated dynamically. Other URLs can be picked up in the standard way by including the site's home page. For example, this include list can be used find all URLs on mysite.com:

https://www.mysite.com
!https://www.mysite.com/sitemap.xml

Notice two things. First that it's okay if there is overlap between the sitemap and other seed URLs in the include list; the spider will remove any duplicates. Also notice that the sitemap line starts with "!". This tells the spider to scan the file sitemap.xml for URLs, but not to include the text in sitemap.xml in the search index.

Enhanced treatment of PDF files

We have upgraded the PDF text extraction engine to handle more character encodings. You should see better retention of punctuation and better sentence construction. (Identifying sentences in PDF files can be challenging because successive lines in a paragraph may not be adjacent in the PDF data.)

Depending on how PDF files are generated, they may not have a title. Titles are important to the search engine as the text is considered highly descriptive of the document. Also, the title is displayed in the search results presented to your visitors. In the new extractor, if there is no title we use a heuristic that chooses the first non-common line of text in the document as the title. Non-common text is text that doesn't appear frequently elsewhere on the website. Common text is usually boiler plate, such as the name of an entity.

For both PDF and HTML files, we recommend that each document have a descriptive title to help the search engine select the document when relevant and to help your visitors understand what the document contains.

Secure Search

Blossom Search can now be integrated with secure websites that use the HTTPS protocol. To use secure search, put https instead of http and ssearch instead of search in the search URL of your search form.

For the results page to be labeled as secure by a browser, you will need to make sure that all hard-coded links in your search results Template File (or Head/Tail files) use HTTPS. Also, if you specify a BASE URL in your Template File, make sure it also uses HTTPS. You can find more information about search Template Files in the Search Guide.

The search results generated by the search engine contain links back to your website based on the search query. Those links match the documents discovered by the search spider during indexing. They are controlled by the Include List associated with the search index. Use the HTTPS protocol for all URLs in the Include List to make the links in the search results secure. (The Include List is set on the Search Configuration page at Blossom.com in the Search Engine Settings section.) If you do change your Include List to use HTTPS, you should also trigger a reindexing to flush documents retrieved via HTTP.

If you need help, send email to Blossom Support.

Using Best Bets to Guide Search Results

Each week you receive a report detailing the search queries of your visitors. Have you taken a look at the most popular queries historically? The queries listed here are good candidates to add to a Best Bets list.

Best Bets allow you to hand-pick pages on your website to appear at the top of the search results for specific queries. Here is how it works:

1. You provide a few must-match terms.
2. You also provide a URL and a short description.
3. When a search includes all of the must-match terms, the URL and description are listed at the top of the search results.

To specify Best Bets, log in to Blossom.com and follow the link Best Bets in the Search Index Settings section:

• The Title will be the heading above the Best Bets. The default is "Best Bets".
• To add a Best Bet, select Add a New Item.
• If a query contains all of the Terms specified, then the Best Bet will be shown. Terms may include the wild card characters "?", to match any one character, and "*" to match zero or more characters.

Redirection program parameters encrypted

Search engines, being complex programs, are a frequent target of hackers looking to exploit a server. A common exploit is to get the search engine to perform an unintended task, such as running an arbitrary system command or getting the server to act as a proxy.

If you use the Blossom adaptive search engine or page highlighting, then you may have noticed a change that prevents a hacker from using the search engine as a proxy. Both adaptation and page highlighting wrap URLs in the search results list with a call to a redirection program. The redirection program retrieves the document at the URL and then does additional processing on the document.

When it was first introduced, the URL for the page to be retrieved was a clear parameter to the redirection program. That attracted hackers wishing to send traffic to websites that appeared to come from Blossom. We will leave it to your ingenuity to figure out ways that might be exploited, but we noticed thousands of redirects daily not coming from search results. Hackers had found an exploit.

To solve the program we now encode the URL to be retrieved. You will see the encoding if you look at the link for the items in a search results list. It is not as informative as having the URL be readable, but it is a lot safer.

By the way, keeping hackers off your server is one nice benefit of using a search service. Besides eliminating the traffic from hacker's probes, should a hacker find an exploit they would still be isolated from your server. It becomes a problem for us, but not for you.

Adaptive ranking of pages in search results

The next revision of the search engine will include the facility to have prior searches influence the order in which documents are listed in the search results. Over time, those pages selected most often by searchers will be rise in the list. To accomplish this, the search engine will keep track of what people select from the search results list; that is, the links in the search results will go indirect through a Blossom search server. Yahoo! search has done this for many years, and Google started doing it within the past couple of years.

Sound interesting? To enable adaptation to your index, just append /adapt to the search URL in your search forms. The links in the search results will then point back to a search server. To prevent hacking, the URLs are encoded.