To test and clean the data without messing with the live CiviCRM installation, I created a separate Drupal+CiviCRM installation to test on.

If your data is unique or complex, importing will not be easy, regardless of the type of software. I was supplied a CSV export file, that I edited in Microsoft Excel and imported into CiviCRM 3.1.4. I could write an entire chapter on cleaning the data and planning the data import, so I will not enter into that here. In short, about the CiviCRM Import Contacts feature I found:

Likes:

• Custom data fields, with multiple values for each field, can be imported in one step from a CSV file.
• You can create a group for each set of imported data, which allows you to view or delete the data you just imported.
• Relationships between and Organization and a Person can be made easily enough, using matching rules.

Dislikes:

• The speed at which contact imports occur is slow.
• The fragility of the import procedure.

Now, more about my dislikes. Regarding the import speed, I'm sure in subsequent versions of CiviCRM this will be improved. What CiviCRM lacks in import speed it makes up for in the vast amount of import features, such as the saving of import field mapping, creation of a group on import, the checking for data errors, and checking for duplicate records.

Regarding the fragility of the import procedure, there could be some trial and error required in order to get your data to import just right. I will address some specific problems I found that are not so straightforward to discover.

1. Strange characters

The result of this example of a "strange" character was that CiviCRM stopped the import of the file on the line directly before this row. Suggested fix: change the character in your data to something acceptable.

2. Rows missing fields

For some reason some rows in the data I was given was missing cells at the end of rows. Excel will not give you any indication the cells are missing, and it took a trick to get Excel to recognize and save them. You will only see the missing fields if you open the CSV file in a text editor, like TextPad, you will see:

The absence of commas at the end of the last two lines shows indications that fields are missing in your CSV file. To fix this, I created a new column in Excel named "Junk", and inserted a space into each cell in that column (use the fill down feature to do this quickly). When importing data in CiviCRM, simply have it ignore this last "Junk" column. In this way, every row in the CSV file will contain the same number of fields.

3. Importing Country and State values

This was a frustrating bug until I found a working method. Organizations that should be located in the United States were importing as "Iraq" or "Suriname". If a row being imported had a State or Province, then the Country value was being ignored, and the State of some other country was selected. I found to combat this error, you can place the Country column of your import file before the other address columns.

With these three odd peculiarities with the CiviCRM Import Contacts feature explained, hopefully this will help others who have similar errors. If it has helped you, or if you have some more tips, your comments are welcome.

Getting shopping cart software to run propertly for a business in Canada is aways difficult. It's a combination of little effort on the developer's part to make it built to work, and what looks like no time spent on testing.

CS Cart gets full points for making it straightforward to create tax rates for different locations - PST for the province, GST for federal taxes, and HST for those three provinces that require it currently (the number of provinces will increase in 2010).

CS Cart does not get full marks for the Canada Post shipping module in CS Cart version 2.0.8. It almost does the minimum required, but it has a bug. I think the Canada Post module is relatively new, so hopefullly this will be improved in the future.

For more than an hour, I was troubleshooting why it would correctly estimate shipping from Canada to Canada, but it would never estimate from Canada to the USA. It always came back with the error message, "Unfortunately no shipping options are available for your location". After digging through the code, I realized that:

1. No matter where the customer was located, the CS Cart always asked for domestic shipping products from Canada Post, and
2. After an hour of troubleshooting and debugging the AJAX enabled shipping estimator, and trying to decipher the session data used by the software, I was no where close to finding the root cause in the CS Cart code.

Having given up looking for the optimal solution, I just hardcoded a fix. The code below was added to the /shippings/can.php file at line 87 (the first line is included here as a reference). It shortcircuits the bad logic to correctly state "if the customer is in the USA, get shipping quotes for the Xpresspost USA product".

	list($header, $result) = fn_http_request('POST', 'http://sellonline.canadapost.ca:30000', $request);

    // Short circuit the code used, because it continues
    //  to use a Canadian shipping code for USA customers
    if($location['country'] == 'US')
    {
      $code = '2030';
    }

Over the previous nine years of NaNoWriMo, enough people have rebelled to warrant a separate special area in the online forums. I would have been a rebel anyway, but it's reassuring to know I'll be joined by others.

I've chosen non-fiction because I want whatever is produced by the end of the month to be something I've always planned to write, even if it is a first draft.

My procrastination to write a book is just as common as the novelist participants; I've always planned to write a book, but it's never gotten off the ground. NaNoWriMo motivates me to attempt to reach the goal and win by writing 50,000 words.

Other things I like:

• Anyone and everyone can be a NaNoWriMo winner
• It only lasts one month
• I'm writing with others, virtually
• I can write anywhere at any time during November
• It costs nothing out of pocket
• The time spent is well spent
• Even if you "lose", there are fringe benefits

Personally, some things I hope to gain:

• Prove my idea is a good one (or disprove it)
• Write a lengthy draft at the fastest speed possible
• Force myself into a daily writing regime, hoping this will ignite my blogging and continuous writing

You may ask, what is my topic? Well, the conversation would probably go something like this,

You: "So you're a non-fiction NaNoWriMo rebel huh? What's your topic?
Me: "Oh... I see what you're after - get your own topic!"

If you've ever planned to write anything - novel or otherwise - consider NaNoWriMo this year.

After:

When you need to understand a new database schema, making a model like this is a big help. I still like DBDesigner4 fromfabFORCE.net to get these simple models done. Yes, I know there's a much newer GUI program, MySQL Workbench, that could do a better job. One day I'll have time to test you again, Workbench. But for me, DBDesigner4 is a simple yet powerful tool.

If you need the date for tomorrow, you can do it in two ways.

The first way is to create the variable in the "code behind", assign it to the template and then you can use it:

In PHP:

$smarty->assign('tomorrow', strtotime('+1 day'));

In Smarty template (or use a date format of your choice):

{$tomorrow|date_format:"%Y-%m-%d"}

The second way is done in the template alone. This is useful if you did not, or did not want to, assign a variable in the code behind. It is also useful if you wanted to use multiple dates (tomorrow, the next day, next week, etc.), or you could even use it in a loop:

In Smarty template:

{math equation=x+y x=$smarty.now y=86400 assign=tomorrow}
{$tomorrow|date_format:"%Y-%m-%d"}

This uses the UNIX timestamp value of $smarty.now, and adds 86,400 seconds to it. Unfortunately you cannot easily add exactly one month - or one year - using this method because each month or year could have a different number of seconds in it.

In future, I plan to say a little more about Smarty and its virtues.

It's a busy day full of chatter on the InterWeb about HTML 5 and its proposed features. "Could it kill Flash and Silverlight", as supposed by Paul Krill at InfoWorld?

One thing is certain: it is not vaporware. After a short search, I found this example implementing a coverflow-style animation from Adnan Osmani (nice work!!) http://blarnee.com/wp/an-improved-javascript-coverflow-demo-using-canvas-and-html5/ If you have an HTML 5 compatible browser - it worked in my Firefox version 3.0.11 - then give it a try. It replicates the animation you'd previously require Flash to do.

My prediction is not can it kill Flash, but when will it kill Flash. Open standards, after gaining momentum, are unstoppable. I'm sure companies like Adobe realize this, and the newer Flash application software will just generate HTML 5 compatible elements. What provides a heavy anchor on progress is how slow users are to upgrade their web browser to be compatible with the new features.

If you don't know what those software programs are, I will not explain here - you can go and read about them. What matters is they are web applications, meaning they run on a web server and you use them in your web browser.

Many computer users - hundreds of millions of them - have from time to time installed software or upgraded software on their own computers. On Microsoft Windows, Mac, and others it's a fairly painless procedure of clicking the "Next" button as you confirm some choices. The computer does some work, you may restart the computer, and 19 times out of 20 you're good to go. Web applications are not that simple.

The general process in upgrading web application software is:

• Backup the files, and the database (basically backup every single thing you can find)
• Delete the old files, which sometimes means all the files
• Upload the new files
• Upload the themes and customizations you had made to your system (sometimes this must be done after running upgrade scripts)
• Run an upgrade script, to upgrade your database. Sometimes, it may even help you merge the customizations you made to templates and files, such as how phpBB3 does).
• Add back all the customizations you may have done (if they were wiped out), and confirm everything is running correctly. You may use some tools to help you merge the new files with the ones you customized.

These usually require any combination of FTP, tar, gzip, zip, ssh, svn, SQL, Linux, and lots of specific knowledge to debug problems that may arise. There's typically no one click install, and if there is, it may work correctly 2 out of 3 times.

The web application software, through its available functions and possible customizations, are very complex. I dread having to upgrade the software because of the possible headaches that arise. But, I don't think it should necessarily any easier to upgrade it. Making it easier would lower the bar of expertise required to upgrade the software, and lowering that bar makes those who should not be upgrading the software themselves to think "it's not that hard, why don't I do it myself".

Unfortunately having a difficult upgrade process causes a steep learning curve for software administrators, and ... it's probably really scary for many people. When a security update is released, they are dependent on hard to find experts to help them.

For example, if you run a hosting server, and a users' domain name is about to expire, would you rather:

1. Send a short email or make a phone call to check it will be renewed by them? Or,
2. Wait until the weekend, the web site may or may not go down. If it goes down, you may be scrambling to try and fix it.

You can think of it in terms of a game of chance. Let's say sending the reminder will take 10% of the time and headache a downtime causes. 50% of the time, the domain name doesn't get renewed, and the downtime is caused. Then that ounce of prevention wins.

It's not magic to keep computers running and keep bugs to a minimum, though to some it seems like the "black arts". Developers and IT professionals experience  much they wish never to repeat again. The good ones will learn from their experiences, and if there's a known cause, next time that situation appears they prevent the bad experience from happening again.

As the years go by, this experience and knowledge starts to gel into some kind of wisdom. Wisdom is not just about fixing all the problems you've seen before , but is more like a "gut feeling" about what is a good choice and what is a not so good choice, and what you should or shouldn't do.

A couple years ago I was discussing this with a someone. Well, I was probably just blathering on about how I had fixed something that I noticed was wrong in some software, but this error was not my responsibility. Funny enough, she has done the same thing before, and for the same reason. We didn't fix the error primarily because we care about the customer. We fixed the errors because when it goes disastrously wrong, we will have to fix it, and it will be far more time consuming and frustrating to fix it than to ensure it never happens in the first place.

In software development, this wisdom works in an ongoing basis. We anticipate how the application will evolve over time. Perhaps we'll add a field in a database table for the date a record was created, because there's a good chance we may need it in the future. I'm sure there are hundreds of ways wisdom allows software and computers to run more smoothly. Unfortunately wisdom cannot be taught, so to gain more wisdom always be one the lookout for ways to anticipate disaster (and prevent it).