This is a very large topic; it is too large to cover in a single blog post. Instead, I'll share what I've done yesterday and today playing with the dataset named "Data.gc.ca Portal Catalogue".

I'm sure it will be fixed soon enough, but the file itself seemed to have character encoding problems. It is in the "latin1" character set, but has unicode characters crammed into it. This caused me hours of grief over two days to diagnose and fix it into something that displayed mostly correct. If you are a MySQL user (or any SQL), this basic utf-8 SQL file should save you the time of trying to import and convert it. data_gc_ca-all_datasets.zip (830KB zip file, 11.5MB expanded)

SQL format was only an intermediary for my plans. As a test, I wanted to load this simple dataset into Solr for searching. You can try the result here:

This does little more than the current Open Data search so far. In order of complexity and infeasibility, the improvements could include:

• Search filtering by category, date, etc.
• It would be similar to filtering auction in Ebay, by clicking links on the left side to refine your search
• Advanced search
• You could search only the French Description field, for example
• Modifications to relevance searching (by popularity), spellchecking, etc.
• Popular datasets would appear at the top of the search results, for example
• Full text search over the entire contents of every dataset
• You can search "Silvicultural", for example, and find dataset 645C42BA-5DC3-412E-AA3C-E37995354CB8, but searching "Clearcutting" (or "coupe à blanc") would also find this dataset, if the .csv file for the dataset was keyword indexed too

Being able to filter by category or date is not possible yet, since the fields do not exist. It would be nice to have  "Date Added" field to indicate when the dataset first appeared on Data.gc.ca, so that newly added datasets could be noticed easily. That would be in addition to adding "Subject Area",  "Creator", and many other fields.

Other issues and questions arise working with this data, which I'll save that for another time.

If you try it out the MySQL table or Solr search, please leave comments with your feedback, questions or errors.

Update January 16, 2012: Thanks to those who attended the presentation at Wordcamp. I have uploaded a revised set of presentation slides in PDF format that includes screen captures of the demonstration portions. If anyone has more questions, please leave comments on this post and I will try and answer them. blog-search-using-solr (PDF; 1.8 MB)

Update January 2012:  I am pleased to be presenting "Blog Search Using Solr" at Wordcamp Victoria 2012. As of today there may still be tickets available, so hurry and get yours now if you haven't already. See you there!

Original topic description: If you discover the default search in WordPress is too basic, you may benefit from installing a Solr for WordPress plugin. Enabling the power of the Lucene searching engine implemented in the Solr server sounds daunting, even to me. The search technology used by Digg, Netflix and Acquia (and others: http://wiki.apache.org/solr/PublicServers) could be yours. As a result of using it, you will discover whether the advantages outweigh the disadvantages. Features like faceted search, keyword highlighting, and more are just the start. Once you see the plugin in action, you will understand the untapped search potential it provides.

You will find this blog using Solr for WordPress right now, and it is still a work in progress (so hopefully it still works when you try it).