Upgrade for search servers in progress

Over the next couple of weeks Blossom will be migrating its search engine to a new hardware architecture to improve processing speed. The change in hardware requires all programs be recompiled. There will also be an upgrade to the underlying operating system.

Ideally, you will not see any change except better performance in generating search results. Please report any anomalies to support@blossoft.com.

Highlighting search snippets and terms

An important feature of Blossom Search is displaying the context of search terms on the search results page. The context, typically a sentence containing the search terms, is called a snippet. To make it clearer why the context is relevant, Blossom Search highlights the search terms in the snippet.

Currently, we are testing highlighting snippets on retrieved HTML files using a new feature of many browsers called "text fragments". The feature is triggered by adding a command at the end of the URL for a document. The command begins with :~:text= and identifies the text to be highlighted. The resulting effect is that the snippet is highlighted and the page is scrolled, if needed, to bring the snippet into view.

A similar feature is being tested on PDF files. Some PDF readers allow search terms to be appended to a URL causing the terms to be highlighted wherever they appear in the document. Again, this feature is not available in all browsers. We've been using Firefox for our tests; Chromium-based browsers do not currently offer this feature.

This feature has left testing and is now live.

Improved handling of JavaScript

 Dynamic websites have long been a challenge for web crawlers because some URLs may be generated dynamically via program code. Website menus, for example, may be created from data tables where the HTML for menu commands and landing pages are assembled at runtime. Without some interpretation of the JavaScript, those landing pages could be missed by a crawler.

The Blossom spider has long scanned JavaScript for potential URLs, but it does not execute JavaScript code. The latest update has enhanced the scan to extract more URLs embedded in strings. As a result, the spider now finds dynamic URLs that were previously missed and thus the number of pages in some indexes has grown.

Upgrade to search software and hardware

 Over the next month the Blossom search server will be migrating to more modern hardware and system software. The changes should be transparent unless you have hard-coded Blossom's IP address anyplace in your system. The result should be better performance and better security. The servers will continue to hosted by Amazon AWS.

In addition to system upgrades, the search software itself will be upgraded. In the past the programs were a mixture of 32 and 64-bit. Moving forward, everything will be 64-bit running on Amazon Linux 2. During this testing period you will see more traffic from Blossom spiders as the new system runs in parallel with the old. The target date for completion is early January.

UPDATE. As of December 26, we have begun the process of moving indexes to the upgraded system. During the week of the 26th, index updates and weekly search reports will be delayed as they are checked for correctness. All indexes should be moved before Jan 1. Please report any anomalies you see to support@blossoft.com.

Expanded sitemap processing

By default, the Blossom spider looks for a sitemap.xml file in each folder it processes. (If desired, this behavior can be turned off from the Search Configuration page.) Recently we discovered that at least one sitemap generator creates a file header that caused our sitemap parser to fail. As a result, some sitemaps were not followed.

The latest spider update has fixed this problem, allowing more sitemaps to be read. Your indexes may now include files that previously were not found.

Upgrade to back-end infrastructure

Work has begun on upgrading the back-end infrastructure supporting Blossom Search, including the customer-facing portal for managing account information and search engine configuration. We will be adding more functionality to the portal, such as a facility for accessing accounting documents and more insight into the contents and use of a search index. Please send suggestions to support@blossoft.com for features you'd like to see.

Robot searches removed from search report

You may have noticed that the number of searches reported in your weekly search report has gone down, perhaps dramatically. We have recently changed the reporting system to ignore searches from known robots.

Robots have long dominated overall web traffic. As a result of their increased sophistication, we've seen an increase in the number of searches carried out by robots. These robot searches skew the search data, making it harder to see just what your (human) visitors are searching for. On your search report the data no longer includes searches from known robots (and you will see the number of robot search that were ignored).

New CSS classes to control search engine output

The appearance of search engine output can be controlled in a variety of ways. The simplest is to use parameters in the search URL, for example to control whether the size or date of a matched page is shown. By using a style sheet you can get control over the fonts, colors, and spacing of the text in the search output. Each component of the output is surrounded by CSS classes. Documentation for using CSS is in the Search Guide.

The next version of the search engine contains a few new classes to control the search output:

Blossom_DocBlock: A div that surrounds all of the output for each document matched by a query. Use this to control block-level behavior such as shading, hovering, and selecting.

Blossom_DocType: If a document is not HTML, then an indicator of the document type is added to the title. This class controls the style of the indicator.

Blossom_MoreButtons: Links for the "next" and "previous" pages of search results are displayed when the "more" option is used. This style controls the format of the link text.

Blossom_SearchForm: A div containing the search-again forms.

You can access the new version of the search engine by using "nquery" in place of "query" in your search URL. Use nquery just for testing, as it changes regularly as we test new search engine features. The changes will migrate to the production search engine, "query", in early November.

Update. These features went live on 11/9/2020. 

New domain names

Blossom Software has used the domain blossom.com since its founding in 1999. During that time we have often received offers to purchase the name. Recently, the company received an offer that could not be refused.

We have taken this opportunity to distribute functionality across a few domains. This improves security and makes the search engine more resistant to attack. Blossom now uses these domain names:

• BlossomSoftware.net for the Blossom Software website. Go here to access your Search Configuration page.
• searchBlossom.com for the Blossom search engine. Use this on your website to invoke the search engine. This domain works for both http and https access.
• blossoft.com for email. For example, use support@blossoft.com to email Blossom Support.

The most important of these changes is the use of searchBlossom.com, as it impacts your search form. search.blossom.com and ssearch.blossom.com will continue to work for a while, but they will eventually be dropped.

First Notice! New URL to access search engine

We have unified access to the search engine for http and https access. Previously, if you used "http://" to call search, you used the server "search.blossom.com". For "https://" you used "ssearch.blossom.com". Using the two servers prevented mixed-content warnings from browsers.

The latest update of the search server can handle both protocols, so two servers are no longer required. Access the new server at searchBlossom.com. Please change your search engine access URLs as follows:

Old: http://search.blossom.com

New: http://searchblossom.com

Old: https://ssearch.blossom.com

New: https://searchblossom.com

This change also affects access to search engine support for popup results:

Old: https://ssearch.blossom.com/search_modal.js

New: https://searchBlossom.com/search_modal.js

Old: https://ssearch.blossom.com/search_modal.css

New: https://searchBlossom.com/search_modal.css

The old URLs will continue to work for several months, but they are scheduled to be retired.

Popup search results

Perhaps the trickiest part of integrating Blossom Search with a website is making the search results page blend in with the style of the site. Blossom has many options for formatting the results page, including head/tail files, template files, style sheets, and XML. Blossom is now testing its second-generation search-results overlay or popup window.

The primary benefit of putting the search results in a popup window is that its layout is independent of the page from which the search is performed and thus there is no need to insert the results into a page template. The popup is created in Javascript and styled in CSS. You can find all of the code needed on the Blossom website. Instructions for using the code are in the Search Guide section titled Using a Modal Popup.

New spider fully deployed

We have completed the transition to the new spider. It has been tested on every index. If a problem was found with your index, your technical contact would have received an email from Blossom Support advising a fix to the problem. In most cases, we were able to implement the fix and your technical contact was just asked to confirm the change.

The new spider has improved handling of dynamic websites and offers some new control over spidering. Among the changes:

• The inclusion/exclusion lists are now more powerful. Read about the full capabilities in the Search Guide.
• Of special note for includes lists is the new $ prefix. It tells the spider to only follow links given in an index file, for example a sitemap file. This is especially useful for richly interconnected sites like blogs.
• The exclude list now allows multiple wildcards (the * character) and an end of URL mark (the $ character). The ? character is not special, following the syntax of robots.txt files.
• Redirections (HTTP codes 301-308) now adhere to include/exclude specifications. Redirections are cached between spidering runs, speeding updates.
• Canonical links are used when possible.
• Cookies are always saved a resent. This improves the experience on session-oriented sites.
• From the search configuration page (at https://blossom.com) you can control the speed of spidering.
• Chunking of document content into logical units (e.g., sentences) has been improved. This is reflected in improved snippets shown search results.

If you see any problems due to the new spider, please let us know by emailing Blossom Support.

Major revision of Blossom spider now being deployed

If you look over the issues discussed in this blog, you'll see that many have arisen due to website content becoming more dynamic. Static web pages are becoming rarer, making the job of spidering more difficult. As a result, we have begun testing a significant rewrite of the Blossom spider.  In addition to handling dynamic pages better, the new spider will offer more flexibility in how sites are traversed. This post will be updated as testing progresses.

 If you monitor your web logs, you may notice extra activity from Blossom as we run the new spider alongside the old. You can pick out visits from Blossom by looking at the User_Agent HTTP header. For the production Blossom spider, the agent is Mozilla/5.0 (Blossom); for the new spider it is Mozilla/5.0 (Blossom/Beta).

We have begun rolling out the new spider to handle the regular update of indexes. In some instances, the new spider may require changes to the configuration of an index. (We will notify your technical contact via email if we make changes for you.) Here are some of the changes we've seen that can impact the contents of an index:

• Stricter handling of redirection URLs. When a request is redirected, either by an HTTP header (e.g. 301 or 302 status code) or by an HTML meta-tag refresh, the redirection URL must satisfy the include/exclude specification for the index.
• Stricter adherence to the HTTP status code and content type as reported by the webserver. Documents will only be added to the index if they are delivered with a status code of 200. HTML pages must either have a content type of text/html or begin with an identifying tag such as or .
• Documents limited to 100MB by default. Likely this will only impact PDF files, and usually just PDFs with lots of images.
• Reading of sitemap.xml and robots.txt are the default.
• Scanning of URLs in javascript strings has been improved. 

New indexer live

A new spidering and indexing engine for Blossom Search is now live with significant changes to improve site coverage and search results. Over time, websites have become much more dynamic with much of the HTML generated at the time of delivery. Dynamic sites present two significant problems:

1. Links to some content may only be generated by client-side programs.
2. Multiple links may generate the same content.

These are not, of course, new concerns, but the increasing complexity of websites makes the spidering and indexing tasks more difficult. The new changes address both of these issues. As a result, you may see your search indexes grow, or perhaps shrink!

The index will grow if your site uses sitemaps. The Blossom spider will now routinely look for sitemap.xml in the root directory for a website. If the file exists, it will use the sitemap to help guide the traversal of a website. If you wish to prevent that behavior, log into the Search Configuration page for your index at Blossom.com, follow the "Spidering, Indexing, and Reporting" link and uncheck the box "Read Sitemaps".

The index will shrink if your site contains or generates very similar pages accessible by different URLs. Duplicate recognition has been improved to pick up pages with nearly identical content regardless of the tag structure. Duplicate pages always have the same title, so using unique titles for unique content will prevent the indexer from ever judging very similar pages to be duplicates.

Use sitemaps to guide Blossom spider

As websites become more dynamic, some links may be generated programmatically rather than specified directly in HTML. While the Blossom spider does search Javascript for URLs embedded in strings, it does not execute Javascript. As a result, URLs generated by string operations can be overlooked.

The spider was recently enhanced to read sitemaps as a way to guide its traversal of a site. By specifying a sitemap in the "include" list for an index, the spider will visit each URL in the sitemap.
 A sitemap is an XML file that lists the URLs on a website. (See https://www.sitemaps.org/ for details.)

For Blossom, the list doesn't have to contain all the URLs on a site; it only needs to include those URLs generated dynamically. Other URLs can be picked up in the standard way by including the site's home page. For example, this include list can be used find all URLs on mysite.com:

https://www.mysite.com
!https://www.mysite.com/sitemap.xml

Notice two things. First that it's okay if there is overlap between the sitemap and other seed URLs in the include list; the spider will remove any duplicates. Also notice that the sitemap line starts with "!". This tells the spider to scan the file sitemap.xml for URLs, but not to include the text in sitemap.xml in the search index.

Enhanced treatment of PDF files

We have upgraded the PDF text extraction engine to handle more character encodings. You should see better retention of punctuation and better sentence construction. (Identifying sentences in PDF files can be challenging because successive lines in a paragraph may not be adjacent in the PDF data.)

Depending on how PDF files are generated, they may not have a title. Titles are important to the search engine as the text is considered highly descriptive of the document. Also, the title is displayed in the search results presented to your visitors. In the new extractor, if there is no title we use a heuristic that chooses the first non-common line of text in the document as the title. Non-common text is text that doesn't appear frequently elsewhere on the website. Common text is usually boiler plate, such as the name of an entity.

For both PDF and HTML files, we recommend that each document have a descriptive title to help the search engine select the document when relevant and to help your visitors understand what the document contains.

Secure Search

Blossom Search can now be integrated with secure websites that use the HTTPS protocol. To use secure search, put https instead of http and ssearch instead of search in the search URL of your search form.

For the results page to be labeled as secure by a browser, you will need to make sure that all hard-coded links in your search results Template File (or Head/Tail files) use HTTPS. Also, if you specify a BASE URL in your Template File, make sure it also uses HTTPS. You can find more information about search Template Files in the Search Guide.

The search results generated by the search engine contain links back to your website based on the search query. Those links match the documents discovered by the search spider during indexing. They are controlled by the Include List associated with the search index. Use the HTTPS protocol for all URLs in the Include List to make the links in the search results secure. (The Include List is set on the Search Configuration page at Blossom.com in the Search Engine Settings section.) If you do change your Include List to use HTTPS, you should also trigger a reindexing to flush documents retrieved via HTTP.

If you need help, send email to Blossom Support.

Using Best Bets to Guide Search Results

Each week you receive a report detailing the search queries of your visitors. Have you taken a look at the most popular queries historically? The queries listed here are good candidates to add to a Best Bets list.

Best Bets allow you to hand-pick pages on your website to appear at the top of the search results for specific queries. Here is how it works:

1. You provide a few must-match terms.
2. You also provide a URL and a short description.
3. When a search includes all of the must-match terms, the URL and description are listed at the top of the search results.

To specify Best Bets, log in to Blossom.com and follow the link Best Bets in the Search Index Settings section:

• The Title will be the heading above the Best Bets. The default is "Best Bets".
• To add a Best Bet, select Add a New Item.
• If a query contains all of the Terms specified, then the Best Bet will be shown. Terms may include the wild card characters "?", to match any one character, and "*" to match zero or more characters.

Redirection program parameters encrypted

Search engines, being complex programs, are a frequent target of hackers looking to exploit a server. A common exploit is to get the search engine to perform an unintended task, such as running an arbitrary system command or getting the server to act as a proxy.

If you use the Blossom adaptive search engine or page highlighting, then you may have noticed a change that prevents a hacker from using the search engine as a proxy. Both adaptation and page highlighting wrap URLs in the search results list with a call to a redirection program. The redirection program retrieves the document at the URL and then does additional processing on the document.

When it was first introduced, the URL for the page to be retrieved was a clear parameter to the redirection program. That attracted hackers wishing to send traffic to websites that appeared to come from Blossom. We will leave it to your ingenuity to figure out ways that might be exploited, but we noticed thousands of redirects daily not coming from search results. Hackers had found an exploit.

To solve the program we now encode the URL to be retrieved. You will see the encoding if you look at the link for the items in a search results list. It is not as informative as having the URL be readable, but it is a lot safer.

By the way, keeping hackers off your server is one nice benefit of using a search service. Besides eliminating the traffic from hacker's probes, should a hacker find an exploit they would still be isolated from your server. It becomes a problem for us, but not for you.

Adaptive ranking of pages in search results

The next revision of the search engine will include the facility to have prior searches influence the order in which documents are listed in the search results. Over time, those pages selected most often by searchers will be rise in the list. To accomplish this, the search engine will keep track of what people select from the search results list; that is, the links in the search results will go indirect through a Blossom search server. Yahoo! search has done this for many years, and Google started doing it within the past couple of years.

Sound interesting? To enable adaptation to your index, just append /adapt to the search URL in your search forms. The links in the search results will then point back to a search server. To prevent hacking, the URLs are encoded.