It’s almost too easy with a Raspberry Pi

No Gravatar

I am part of Overskrift.dk – a local social media monitoring service built on a LAMP stack. We are the server itself, but also quite a few older laptops. They do have a couple of downsides though:

  1. Big (-ish)
  2. Clunky
  3. Noisy
  4. Power consuming
  5. Have mechanical harddisks


They each need PHP, cUrl and a MySQL client library installed in order to function – but then they are actually able to support our aggregation tasks quite nicely.

So the other day when a harddisk crashed, I came to think of the Raspberry Pi (popularly dubbed the $25 computer) – It actually set me back DKK 638,- including P&P from a danish webshop (a little more over $100), but that was only because I insisted on a cabinet, a power supply and an SD card. Still I can get three of these for the same price as 1 very low-end netbook. An our after the postman delivered the envelope, it was up and running aggregating away.

Raspberry Pi

From my laptop I downloaded the Raspian “wheezy” SD card image – it downloads a .zip file to unzip into a .img-file. On Mac and Linux the image can easily be copied to the SD card (but taking about 20 minutes). I used this process.

Once the image was downloaded I moved the SD card to theRaspberry Pi unit, plugged a keyboard into one of the USB ports, connected my TV through the HDMI connector and powered up. First boot took about 30 seconds and took me directly to a very nice configuration UI, I set the locale and timezone and asked the ssh-daemon to start on boot.

raspi-config
The Raspi Config UI

Next step was to shut down and move the box over to a LAN connection by my modem. Now only the LAN connection and the power supply was connected.

Coming from the Ubuntu world, installing PHP, cUrl and the MySQL client libraries was a question of logging on,  running

sudo apt-get update
sudo apt-get install php5 curl php5-curl php5-mysql mysql-client

Now I could simply copy my PHP code to the filesystem and set up cron jobs just as I would in Ubuntu.

UPDATE 2014-02-21: It has been almost a year since we started using Raspberry PI’s for our aggregation purposes. Since then, we’ve added a couple of pi’s for this specific purpose and retired all other aggregations machines, probably saving quite a bit on our power bill.

 

Ubuntu 10.4 – up with only one quirk so far

No Gravatar

This morning I let my Ubuntu 9.10 upgrade itself to 10.4. The upgrade itself ran fine with only a few interruptions where I had to confirm or deny changes to configuration files for PHP etc.

As I landed at the office and hooked up the laptop (Thinkpad T61P) however, my external monitor was flickering a lot. Tried booting into Windows and everything was fine. Google’d and found this post on Ubuntu Forums.

The post suggests making changes to menu.lst in /boot/grub. This is a somewhat short term solution, because the problem will probably reappear at the next kernel upgrade (which could be just around the corner). The fix may be on it’s way in an actual Ubuntu fix, but i chose a different approach which may also seem hack’ish – the fix could be more robust though.

Disclaimer: If you are not sure what you are doing, be very careful. Misspellings or deviations from the description below may lead to an unbootable system. This worked for me and doesn’t have to work for you. Make sure to back up any files you change.

OK? Let’s go..

First start a terminal (ALT-F2, enter “gnome-terminal”):

cd /etc/grub.d

Then:

sudo nano 10_linux

which will open an editor. Around line 96 inside the linux_entry-function is a line that looks like this:

linux    ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}

and it should be changed to:

linux    ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro radeon.modeset=0 ${args}

Having done this, you should ask grub to rebuild it’s configuration files with:

sudo update-grub

Reboot and go on without the flicker..

Will ideas of the Massachusetts law spread?

No Gravatar

On reddit, I found this article regarding a new law i Massachusetts that is to increase the security of personally identifiable information.

If you have personally identifiable information (PII) about a Massachusetts resident, such as a first and last name, then you have to encrypt that data on the wire and as it’s persisted. Sending PII over HTTP instead of HTTPS? That’s a big no no. Storing the name of a customer in SQL Server without the data being encrypted?  No way, Jose. You’ll get a fine of $5,000 per breach or lost record. If you have a database that contains 1,000 names of Massachusetts residents and lose it without the data being encrypted that’s $5,000,000. Yikes.

The idea is good (yay – better security for anyone registering on a website) and bad (expensive). As a consultant in a country not covered by the law I could care less, but is this in general a good idea? Securing data and sending sensitive information across secure connections is always a good idea, and it could hit the EU soon. Documenting how you secure the Personally Identifiable Information (PII)? Who can argue with that being a good idea? As an IT consultant you won’t hear me cry myself to sleep.

But the proportions of the legislation seem unclear. If you log on to my website (securely of course), and I feed you the following pages through clear text HTTP – can I be fined if your name appear on the page as in “Reader XYZ is logged in”?

I guess so.

It strikes me that there is no level of sensitivity defined – anything considered personal must be secured. As a legislator, it seems very easy to do this when you don’t have to pick up the bill.

If this kind of legislation should hit Europe, I hope someone would elaborate a bit on the do’s and dont’s:

  • Are all kinds of data included? Can I be fined if you, as a Massachusetts citizen, post a comment to this and include your name? (the database is secured, but nothing is sent through HTTPS)
  • Would it make sense to allow users to wave their rights, and thereby allow users to work with online applications that are not 100% secure?

Karmic Koala – post install notes

No Gravatar

Installed Ubuntu 9.10 a few days ago. Everything seemed OK (very very nice actually) except for a little network performance glitch that may also bother you.√Ǭ† Canonical has once again taken Linux one step further away from the status of geek-challenge to a user friendly alternative to other OS’es.

One thing bothered me though – general network responsiveness seemed seriously degraded. Digging around i found that IPv6 had taken precedence to IPv4 in a few ways – one being DNS lookup sequence. Apparently all lookups was attempted through IPv6 first. My router and network is in no way configured for IPv6 and therefore every connection-attempt to uncached hosts would have to wait for the IPv6 timeout. Two things gave me responsiveness back.

  1. Disable IPv6 at the OS level
  2. Disable IPv6 in Firefox (should give more responsiveness for any OS not running on an IPv6 network)

Maybe you would think that the first step would be enough, but Firefox seems more responsive after step 2. Follow this step to disable IPv6 in Firefox.

Ubuntu 9.10 introduces the GRUB2 bootloader on clean installations only. The upgrade process from, say version 9.04, will not upgrade GRUB and will leave you with the previous version.

To disable IPv6 OS wide, i followed this sequence and voila – responsiveness returned to my netbook (and soon to my other Ubuntu installations).

2009-11-22 update: Had to downgrade to 9.04 again – No matter what I did, which forums I visited – I could not get my Huawei 3G Modem to connect.

Netbooks – the necessary new design test-tool

No Gravatar

Writing this on my Lenovo S10e netbook, I am furious. It’s before mid-day and yet I have had two experiences of software designs that did not consider netbooks a platform – or at least the new low screen resolution these computers imply. And before you call me a whiner (besides the fact that you’d be somewhat right), this is just a description of the changes I will make to include netbook users as an audience of software and website in the future.

Whine #1: Twice I’ve been cripled by software that saw the low screen resolution as a handicap – one of them on purpose. First I installed Pidgin – the cool cross-platform/cross-protocol IM and IRC client. I like it a lot, but on Windows some dialogs are too big and will not allow me to navigate to the OK/Cancel buttons at the bottomof the settings dialog. Fortunately this is Open Source stuff, so I can just participate and actively fix this myself.

Whine #2: I had to install a printer driver for my HP Photosmart 2575 printer – the install took over one hour because of some “ingenious” package system. That obviously poor user experience decision aside – the minimum requirement for the printer driver is a screen resolution of 800×600 pixels. My S10e runs at 1024 x 576. The consequence – I cannot print from my netbook in Windows because the printer driver won’t finish installing, as it has an irrelevant requirement. Fortunately I am dual-booting with Ubuntu, which has excellent support for my printer (without the requirement)

I’ve read somewhere that 20% of all computers that will be sold in 2009 will be netbooks. Some producers (including Asus) will stop production of 8.9″ -screen netbooks. The 10-inchers seems to dominate right now, and probably for the rest of the year (note: my guess only). Every company designing software will have to take this into account before they ship the next version of any product with a user interface.

I’ve worked on so many web projects with art directors saying: “Nobody runs 640×480 or 800×600 anymore”. Hey we know – but do you know how many users run with their browser maximized because a designer thought up a design that required it? Web designers don’t own the real estate of the users screen resolution – they can only hope to own the area the browser is sized to – and you have to count on users having at least one open history/bookmarks sidebar and/or plug-in and/or Google/MSN/Web developer toolbar enabled.

My point should be rather obvious: It is vital to include netbooks as testplatforms for serious UI-designers or your product will no longer be compatible with the equipment of the customers you want. The rules have changed – live with it.

You may argue that netbook users are asking for it, but if the competing website or software support the netbooks and you don’t – the users is no longer making the decision of leaving you.

Netbooks growing up

No Gravatar

ideapad s10_red

This post is written on my new netbook – a Lenovo ideapad s10e. Actually I had given up on netbooks. My first one was an Asus EEE PC 900 with SSD and that machine was a serious disappointment. I have always liked Asus hardware, mainly because it is extremly well updated with software. That was also the case with the eee 900, but in order to make it cheap enough a cut was made on the CPU and on the keyboard. Actually I think I could live with the somewhat slow Dothan CPU (Celeron’ish), but combined with the poor keyboard the experience was like:

I don’t know if the machine refuses to react on my keypress because the stroke wasn’t registered on the keyboard or because I am waiting for the CPU.

That is of course not acceptable. So the too-cheap netbook has been gathering dust for a while now.

Some colleagues also purchased netbooks. One of them a Medion Akoya (looks very much like the MSI Wind – which 95% of the hardware probably is). It had an Intel Atom processor, which seems to do A LOT for these small laptops. The price did get an extra nudge compared to the Asus eee PC 900, but that can actually mean the difference between a usable and an unusable netbook.

The Lenovo machine is Atom-based, responsive and so far a really really nice piece of machinery. I dual-boot Windows and Ubuntu Linux on it and have yet to encounter missing drivers or missing responsiveness from the keyboard.

As with anything, you get what you pay for and you actually don’t have to pay that much more for a netbook that actually behaves as you would expect.

Ubuntu two-finger scrolling

No Gravatar

My brother is now a fairly long time Mac-user. I haven’t gone down that road (yet). Last night he was here and i showed him my Asus EEE (which is currently running Ubuntu 8.10).

“Cool” – he said. “It supports two-finger-scrolling”. I didn’t know, but he is right. Googling for it found me this. Apparently the touchpad is new enough in the Asus EEE to support two finger scrolling. Nice to find hidden coolness.

Magpie RSS/Snoopy problem finally solved

No Gravatar

MagpieRSS IS great – I like it a lot, and it almost does exactly what you need to aggregate RSS feeds from PHP. MagpieRSS has one achilles heel in Snoopy. MagpieRSS relies on Snoopy as the HTTP-client (browser-component) that fetches feeds from websites – be it RSS or Atom formatted – It fetches files from a webserver.

The latests Snoopy release was in 2005 (as was the latest MagpieRSS), which could indicate that the development community has abandoned the code. The problem with Snoopy is that it has low tolerance to the use of carriage-return and linefeed characters in the header of HTTP responses from webservers. For some requests that means certain files, though rich in content, appear empty if they are fetched using Snoopy.

At first I tried downloading the latest Snoopy release from Sourceforge. It seemed to do the trick for some feeds, but aparently not all. Arnold Daniels came to my rescue. He has devised a wrapper – on the “outside” it appears to be Snoopy – but on the inside PHP’s Curl library has taken the place of a lot of the original Snoopy code. Give his alternative library a spin if you are using MagpieRSS for anything the should be somewhat resistant to deviations from standards.

Generate PDF files from PHP

No Gravatar

For a web project I needed to create invoices, to be downloaded or e-mailed to clients in PDF format. The platform is PHP and the documentation for this almost exclusively describe PDFLib for this purpose. I am sure PDFLib is fine, and the tutorials that come with the package looks really good.

But..

You pay for PDFLib – I don’t mind paying for software, you should pay for usable high-quality stuff, which I am sure PDFLib is. US$ 995 is a little steep though – compared to free. The FAQ however mentions a few free alternatives. Right now I am looking into FPDF – which seem to do the trick.

Actually I found out that my previous ISP (invoicing their customers only by PDF invoices via E-mail) use FPDF. This tells me that FPDF is more than ready for production. I’ll let you know if I something disappoints me – although I would hate to bash a truly free initiative such as this one. Oh – and by the way it has another edge to PDFLib – It is written in PFP and does not need server reconfiguration to work. If you want a head start, there are quite an extensive list of code examples available.

MySQL query/indexing tip

No Gravatar

I originally learned about this MySQL shortcoming in the book High Performance MySQL, and luckily I remembered it for an issue that struck me today.

I have two tables – one containing blog-posts, and one containing a record for each tag on these posts. Now the requirement was to show a blog post AND to show the five latest related posts. Related posts in this scenario are other posts that share one or more tags with this post.

Now, it does not take one second to find a blog post, and it does not take one second to find tags related to it. And actually it does not even take a second to find a list of posts having a list of tags. The reason why all these requests take less than a second is because the tables have been supplied with efficient indexes – the table of blogposts has more than 3.5 million records and there are about 700.000 tags.

The problem was that when we tried to generate a list of related posts, the query took up to twelve seconds to run. It sucked and looked like this:

SELECT distinctrow i.id, i.item_title, i.item_url
FROM rss_items_master i
INNER JOIN rss_item_tag t ON i.id=t.item_id
WHERE t.item_tag IN (SELECT item_tag FROM rss_item_tag WHERE item_id=$itemid)
AND t.item_id != $itemid
ORDER BY i.item_date DESC
LIMIT 5;

In plain english we request a list of items that does not match the specific post, but they must have at least one tag that match one of the tags from the specific post. The nested SELECT-statement produce a list of tags to look for, and the outer SELECT statement pulls blog posts containing tags found in the nested SELECT. This does not perform, because only one index can be used per query – so either the index from the rss_item_tag table or from the rss_items_master table is ignored (i did not bother figuring out which one was ignored). To fix it, I split the SQL in two (PHP example):

$sql_tags = "SELECT item_tag FROM rss_item_tag WHERE item_id=$itemid;";
$arr_tags = query_array($sql_tags, $link);
// $tag_o will be an array of tags from the original post.
$tag_o = Array();
foreach($arr_tags as $tag_i) {
array_push($tag_o, mysql_real_escape_string($tag_i[0]));
}
$tags = implode("','", $tag_o);

Now $tags contain a list og tags to look for – actually it contains exactly what the nested SELECT-part did in the slow query. Now these two queries will actually complete in less than a second:

select distinctrow i.id, i.item_title, i.item_url
from rss_items_master i inner join rss_item_tag t on i.id=t.item_id
and t.item_tag in ('$tags')
and t.item_id != $itemid
order by i.item_date desc
limit 5;

So even though the SQL has been split in two, and even though we are passing a lot of information through variables that we have to build from the first query and added quite a lot of code – this outperforms the first single query by a factor larger than 10 – simply because it enables MySQL to use the indexes it should.

For more MySQL performance tips, check out MySQL Performance Blog.