This morning I let my Ubuntu 9.10 upgrade itself to 10.4. The upgrade itself ran fine with only a few interruptions where I had to confirm or deny changes to configuration files for PHP etc.

As I landed at the office and hooked up the laptop (Thinkpad T61P) however, my external monitor was flickering a lot. Tried booting into Windows and everything was fine. Google’d and found this post on Ubuntu Forums.

The post suggests making changes to menu.lst in /boot/grub. This is a somewhat short term solution, because the problem will probably reappear at the next kernel upgrade (which could be just around the corner). The fix may be on it’s way in an actual Ubuntu fix, but i chose a different approach which may also seem hack’ish – the fix could be more robust though.

Disclaimer: If you are not sure what you are doing, be very careful. Misspellings or deviations from the description below may lead to an unbootable system. This worked for me and doesn’t have to work for you. Make sure to back up any files you change.

OK? Let’s go..

First start a terminal (ALT-F2, enter “gnome-terminal”):

cd /etc/grub.d

Then:

sudo nano 10_linux

which will open an editor. Around line 96 inside the linux_entry-function is a line that looks like this:

linux    ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}

and it should be changed to:

linux    ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro radeon.modeset=0 ${args}

Having done this, you should ask grub to rebuild it’s configuration files with:

sudo update-grub

Reboot and go on without the flicker..

On reddit, I found this article regarding a new law i Massachusetts that is to increase the security of personally identifiable information.

If you have personally identifiable information (PII) about a Massachusetts resident, such as a first and last name, then you have to encrypt that data on the wire and as it’s persisted. Sending PII over HTTP instead of HTTPS? That’s a big no no. Storing the name of a customer in SQL Server without the data being encrypted?  No way, Jose. You’ll get a fine of $5,000 per breach or lost record. If you have a database that contains 1,000 names of Massachusetts residents and lose it without the data being encrypted that’s $5,000,000. Yikes.

The idea is good (yay – better security for anyone registering on a website) and bad (expensive). As a consultant in a country not covered by the law I could care less, but is this in general a good idea? Securing data and sending sensitive information across secure connections is always a good idea, and it could hit the EU soon. Documenting how you secure the Personally Identifiable Information (PII)? Who can argue with that being a good idea? As an IT consultant you won’t hear me cry myself to sleep.

But the proportions of the legislation seem unclear. If you log on to my website (securely of course), and I feed you the following pages through clear text HTTP – can I be fined if your name appear on the page as in “Reader XYZ is logged in”?

I guess so.

It strikes me that there is no level of sensitivity defined – anything considered personal must be secured. As a legislator, it seems very easy to do this when you don’t have to pick up the bill.

If this kind of legislation should hit Europe, I hope someone would elaborate a bit on the do’s and dont’s:

• Are all kinds of data included? Can I be fined if you, as a Massachusetts citizen, post a comment to this and include your name? (the database is secured, but nothing is sent through HTTPS)
• Would it make sense to allow users to wave their rights, and thereby allow users to work with online applications that are not 100% secure?

Installed Ubuntu 9.10 a few days ago. Everything seemed OK (very very nice actually) except for a little network performance glitch that may also bother you.  Canonical has once again taken Linux one step further away from the status of geek-challenge to a user friendly alternative to other OS’es.

One thing bothered me though – general network responsiveness seemed seriously degraded. Digging around i found that IPv6 had taken precedence to IPv4 in a few ways – one being DNS lookup sequence. Apparently all lookups was attempted through IPv6 first. My router and network is in no way configured for IPv6 and therefore every connection-attempt to uncached hosts would have to wait for the IPv6 timeout. Two things gave me responsiveness back.

1. Disable IPv6 at the OS level
2. Disable IPv6 in Firefox (should give more responsiveness for any OS not running on an IPv6 network)

Maybe you would think that the first step would be enough, but Firefox seems more responsive after step 2. Follow this step to disable IPv6 in Firefox.

Ubuntu 9.10 introduces the GRUB2 bootloader on clean installations only. The upgrade process from, say version 9.04, will not upgrade GRUB and will leave you with the previous version.

To disable IPv6 OS wide, i followed this sequence and voila – responsiveness returned to my netbook (and soon to my other Ubuntu installations).

2009-11-22 update: Had to downgrade to 9.04 again – No matter what I did, which forums I visited – I could not get my Huawei 3G Modem to connect.

Writing this on my Lenovo S10e netbook, I am furious. It’s before mid-day and yet I have had two experiences of software designs that did not consider netbooks a platform – or at least the new low screen resolution these computers imply. And before you call me a whiner (besides the fact that you’d be somewhat right), this is just a description of the changes I will make to include netbook users as an audience of software and website in the future.

Whine #1: Twice I’ve been cripled by software that saw the low screen resolution as a handicap – one of them on purpose. First I installed Pidgin - the cool cross-platform/cross-protocol IM and IRC client. I like it a lot, but on Windows some dialogs are too big and will not allow me to navigate to the OK/Cancel buttons at the bottomof the settings dialog. Fortunately this is Open Source stuff, so I can just participate and actively fix this myself.

Whine #2: I had to install a printer driver for my HP Photosmart 2575 printer – the install took over one hour because of some “ingenious” package system. That obviously poor user experience decision aside – the minimum requirement for the printer driver is a screen resolution of 800×600 pixels. My S10e runs at 1024 x 576. The consequence – I cannot print from my netbook in Windows because the printer driver won’t finish installing, as it has an irrelevant requirement. Fortunately I am dual-booting with Ubuntu, which has excellent support for my printer (without the requirement)

I’ve read somewhere that 20% of all computers that will be sold in 2009 will be netbooks. Some producers (including Asus) will stop production of 8.9″ -screen netbooks. The 10-inchers seems to dominate right now, and probably for the rest of the year (note: my guess only). Every company designing software will have to take this into account before they ship the next version of any product with a user interface.

I’ve worked on so many web projects with art directors saying: “Nobody runs 640×480 or 800×600 anymore”. Hey we know – but do you know how many users run with their browser maximized because a designer thought up a design that required it? Web designers don’t own the real estate of the users screen resolution – they can only hope to own the area the browser is sized to – and you have to count on users having at least one open history/bookmarks sidebar and/or plug-in and/or Google/MSN/Web developer toolbar enabled.

My point should be rather obvious: It is vital to include netbooks as testplatforms for serious UI-designers or your product will no longer be compatible with the equipment of the customers you want. The rules have changed – live with it.

You may argue that netbook users are asking for it, but if the competing website or software support the netbooks and you don’t – the users is no longer making the decision of leaving you.

This post is written on my new netbook – a Lenovo ideapad s10e. Actually I had given up on netbooks. My first one was an Asus EEE PC 900 with SSD and that machine was a serious disappointment. I have always liked Asus hardware, mainly because it is extremly well updated with software. That was also the case with the eee 900, but in order to make it cheap enough a cut was made on the CPU and on the keyboard. Actually I think I could live with the somewhat slow Dothan CPU (Celeron’ish), but combined with the poor keyboard the experience was like:

I don’t know if the machine refuses to react on my keypress because the stroke wasn’t registered on the keyboard or because I am waiting for the CPU.

That is of course not acceptable. So the too-cheap netbook has been gathering dust for a while now.

Some colleagues also purchased netbooks. One of them a Medion Akoya (looks very much like the MSI Wind – which 95% of the hardware probably is). It had an Intel Atom processor, which seems to do A LOT for these small laptops. The price did get an extra nudge compared to the Asus eee PC 900, but that can actually mean the difference between a usable and an unusable netbook.

The Lenovo machine is Atom-based, responsive and so far a really really nice piece of machinery. I dual-boot Windows and Ubuntu Linux on it and have yet to encounter missing drivers or missing responsiveness from the keyboard.

As with anything, you get what you pay for and you actually don’t have to pay that much more for a netbook that actually behaves as you would expect.

My brother is now a fairly long time Mac-user. I haven’t gone down that road (yet). Last night he was here and i showed him my Asus EEE (which is currently running Ubuntu 8.10).

“Cool” – he said. “It supports two-finger-scrolling”. I didn’t know, but he is right. Googling for it found me this. Apparently the touchpad is new enough in the Asus EEE to support two finger scrolling. Nice to find hidden coolness.

MagpieRSS IS great – I like it a lot, and it almost does exactly what you need to aggregate RSS feeds from PHP. MagpieRSS has one achilles heel in Snoopy. MagpieRSS relies on Snoopy as the HTTP-client (browser-component) that fetches feeds from websites – be it RSS or Atom formatted – It fetches files from a webserver.

The latests Snoopy release was in 2005 (as was the latest MagpieRSS), which could indicate that the development community has abandoned the code. The problem with Snoopy is that it has low tolerance to the use of carriage-return and linefeed characters in the header of HTTP responses from webservers. For some requests that means certain files, though rich in content, appear empty if they are fetched using Snoopy.

At first I tried downloading the latest Snoopy release from Sourceforge. It seemed to do the trick for some feeds, but aparently not all. Arnold Daniels came to my rescue. He has devised a wrapper – on the “outside” it appears to be Snoopy – but on the inside PHP’s Curl library has taken the place of a lot of the original Snoopy code. Give his alternative library a spin if you are using MagpieRSS for anything the should be somewhat resistant to deviations from standards.

For a web project I needed to create invoices, to be downloaded or e-mailed to clients in PDF format. The platform is PHP and the documentation for this almost exclusively describe PDFLib for this purpose. I am sure PDFLib is fine, and the tutorials that come with the package looks really good.

But..

You pay for PDFLib – I don’t mind paying for software, you should pay for usable high-quality stuff, which I am sure PDFLib is. US$ 995 is a little steep though – compared to free. The FAQ however mentions a few free alternatives. Right now I am looking into FPDF – which seem to do the trick.

Actually I found out that my previous ISP (invoicing their customers only by PDF invoices via E-mail) use FPDF. This tells me that FPDF is more than ready for production. I’ll let you know if I something disappoints me – although I would hate to bash a truly free initiative such as this one. Oh – and by the way it has another edge to PDFLib – It is written in PFP and does not need server reconfiguration to work. If you want a head start, there are quite an extensive list of code examples available.

I originally learned about this MySQL shortcoming in the book High Performance MySQL, and luckily I remembered it for an issue that struck me today.

I have two tables – one containing blog-posts, and one containing a record for each tag on these posts. Now the requirement was to show a blog post AND to show the five latest related posts. Related posts in this scenario are other posts that share one or more tags with this post.

Now, it does not take one second to find a blog post, and it does not take one second to find tags related to it. And actually it does not even take a second to find a list of posts having a list of tags. The reason why all these requests take less than a second is because the tables have been supplied with efficient indexes – the table of blogposts has more than 3.5 million records and there are about 700.000 tags.

The problem was that when we tried to generate a list of related posts, the query took up to twelve seconds to run. It sucked and looked like this:

SELECT distinctrow i.id, i.item_title, i.item_url
FROM rss_items_master i
INNER JOIN rss_item_tag t ON i.id=t.item_id
WHERE t.item_tag IN (SELECT item_tag FROM rss_item_tag WHERE item_id=$itemid)
AND t.item_id != $itemid
ORDER BY i.item_date DESC
LIMIT 5;

In plain english we request a list of items that does not match the specific post, but they must have at least one tag that match one of the tags from the specific post. The nested SELECT-statement produce a list of tags to look for, and the outer SELECT statement pulls blog posts containing tags found in the nested SELECT. This does not perform, because only one index can be used per query – so either the index from the rss_item_tag table or from the rss_items_master table is ignored (i did not bother figuring out which one was ignored). To fix it, I split the SQL in two (PHP example):

$sql_tags = "SELECT item_tag FROM rss_item_tag WHERE item_id=$itemid;";
$arr_tags = query_array($sql_tags, $link);
// $tag_o will be an array of tags from the original post.
$tag_o = Array();
foreach($arr_tags as $tag_i) {
array_push($tag_o, mysql_real_escape_string($tag_i[0]));
}
$tags = implode("','", $tag_o);

Now $tags contain a list og tags to look for – actually it contains exactly what the nested SELECT-part did in the slow query. Now these two queries will actually complete in less than a second:

select distinctrow i.id, i.item_title, i.item_url
from rss_items_master i inner join rss_item_tag t on i.id=t.item_id
and t.item_tag in ('$tags')
and t.item_id != $itemid
order by i.item_date desc
limit 5;

So even though the SQL has been split in two, and even though we are passing a lot of information through variables that we have to build from the first query and added quite a lot of code – this outperforms the first single query by a factor larger than 10 – simply because it enables MySQL to use the indexes it should.

For more MySQL performance tips, check out MySQL Performance Blog.

I was looking for an XML editor to perform a few simple tasks. Since i worked heavily with XML a few years ago, the Windows XML editor of choice is apparently still the Altova XML Spy. You can get it to do anything with XML (except – i think – transform XML to coffee). Negative minds would probably call it bloated and expensive. Positive minds would look for a cheap agile alternative.

Using the Open Source Alternative-site it is possible to find applications doing somewhat the same as commercial products. For the XML Spy query, the result is XML Copy Editor – a SourceForge application that has currently reached version 1.0.9. My needs were met as it:

1. color-codes XML
2. beautify/pretty-print XML with consistent indentations to increase readability
3. opens files directly from URL’s

Also it XSLT transforms and has a small set of other features you may or may not need. Before you venture into extensive license bill-paying, you may try out this nifty little XML editor.

I am a Wiki-virgin. Not in the sense of being a wiki-reader/contributor, but today I set up my first Wiki which is about to go online. Wiki’s based on the MediaWiki software (which is also used for WikiPedia) are actually skinable, but as content usually is king in Wiki land, very few operators make an effort to change skin before going online. The well known default Monobook skin is used almost everywhere (you know – the one where the monochrome image of an book is the back-drop).

This has one obvious benefit: People recognize it as being a Wiki-type site immediately. Indeed a strong argument to keep things as they are. I may be running my mouth of as the green first day Wiki operator, and maybe I have actually used Wiki based sites that I did not recognize because it used some great unknown skin. But doesn’t the Wiki deserve it’s Kubrick skin – which was considered the defacto blog-look a few years back. People would look at the skin (which is beautiful) and say “Ahh – that’s a weblog”. These days I (and probably a few other) say “Yuck – It’s a Wiki”.

Michael Heilemann did such a great job with Kubrick. Isn’t it time for Wiki’s to have their own updated slick skin? Do you have it or know where to get it? There is one Wiki-skin I like: Cavendish. I would love to be able to create one myself, but my graphic skills are unfortunately only focused around my eyes – not in my hands.

I am in the middle of a project with a client – a project to improve their time usage registration procedures. They had been looking at several commercial solutions – all requiring customizations – most of them costing “an arm and a leg”.

They actually started a project with one vendor who thought they could deliver, but apparently forgot to listen for client needs and forgot to set client expectations correctly – the project ran for six months and finally they had to retreat and stop the project in progress because they could not deliver.

I got a second chance to present my idea which is now running and in further development. It is based on the Open Source project Achievo, originally developed and opened up by the cool guys at iBuildings in the Netherlands.

The advancements of the processes around time registration and tracking is usually “evolutionary” for companies like my client. They know they need to register time in order to bill the client at the end of the month. In this case, the consultants would initially share an Excel sheet in which they would register their hours. It worked, but apart from being limited to only one person entering hourly usage at a time, it was very difficult to verify that everybody had remembered to register all billable hours. And that may or may not be quite expensive in the end – they actually didn’t know.

Achievo is based on modules and it has a billing module which I guess is used by more or less anyone running the system. It will be included as a standard module in the next release. My main task has been to integrate with my clients CRM system (regularly import customers and create dummy projects for them). Also I created a few reports to ease the actual billing, as my client cannot use the billing module as-is. They have a somewhat special price matrix, which I needed to get Achievo in tune with. It was also necessary to build some tidbits that would enable handling of prepaying clients bying a predefined number of hours at special rates. I hope that I will dig up some time to feed some business process documentation and project templates back to the Achievo community.

Achievo runs on a LAMP stack (Linux, Apache, MySQL and PHP). Check out the demo sites running version 1.2.1 (the current release) and the nightly builds for the upcoming version 1.3. My one-man company can use Achievo 100% as is. My client – a 25 person consultancy firm – has been running Achievo since the beginning of this year and we are just now entering phase two, where we will add quite a few more “need” and “nice” features.