It’s almost too easy with a Raspberry Pi

No Gravatar

I am part of Overskrift.dk – a local social media monitoring service built on a LAMP stack. We are the server itself, but also quite a few older laptops. They do have a couple of downsides though:

  1. Big (-ish)
  2. Clunky
  3. Noisy
  4. Power consuming
  5. Have mechanical harddisks


They each need PHP, cUrl and a MySQL client library installed in order to function – but then they are actually able to support our aggregation tasks quite nicely.

So the other day when a harddisk crashed, I came to think of the Raspberry Pi (popularly dubbed the $25 computer) – It actually set me back DKK 638,- including P&P from a danish webshop (a little more over $100), but that was only because I insisted on a cabinet, a power supply and an SD card. Still I can get three of these for the same price as 1 very low-end netbook. An our after the postman delivered the envelope, it was up and running aggregating away.

Raspberry Pi

From my laptop I downloaded the Raspian “wheezy” SD card image – it downloads a .zip file to unzip into a .img-file. On Mac and Linux the image can easily be copied to the SD card (but taking about 20 minutes). I used this process.

Once the image was downloaded I moved the SD card to theRaspberry Pi unit, plugged a keyboard into one of the USB ports, connected my TV through the HDMI connector and powered up. First boot took about 30 seconds and took me directly to a very nice configuration UI, I set the locale and timezone and asked the ssh-daemon to start on boot.

raspi-config
The Raspi Config UI

Next step was to shut down and move the box over to a LAN connection by my modem. Now only the LAN connection and the power supply was connected.

Coming from the Ubuntu world, installing PHP, cUrl and the MySQL client libraries was a question of logging on,  running

sudo apt-get update
sudo apt-get install php5 curl php5-curl php5-mysql mysql-client

Now I could simply copy my PHP code to the filesystem and set up cron jobs just as I would in Ubuntu.

UPDATE 2014-02-21: It has been almost a year since we started using Raspberry PI’s for our aggregation purposes. Since then, we’ve added a couple of pi’s for this specific purpose and retired all other aggregations machines, probably saving quite a bit on our power bill.

 

Workaround to Facebook’s Graph API Search locale change

No Gravatar

So Facebook decided to change the meaning of the locale parameter to the search parts of the Graph API – and label the decision “by design” after 2 months of thinking. Not “design change” or “bug we don’t feel like fixing” – just “If it worked before – that was a mistake”. The handling of the problem is described in this bug-report.

If you do not use it, you probably don’t care, but if you do this would probably be fatal to your searches.

The problem

I would like to know what Facebook users in my country (Denmark) writes about my local branch of the phone company Telia. To find out, I would ask the graph:

https://graph.facebook.com/search?q=Telia&locale=da_DK

The locale-parameter would ensure that I only get results from danish Facebook sources. At least that was the case until December 5, 2012. After that, the locale-parameter changed role – now it will just localize certain field identifiers in the JSON reply. This is useless and all of a sudden I get heaps of irrelevant results from users in other countries in other languages. Facebook remains silent about what to do instead.

The workaround

For my projects I’ve used the following workaround – it’s Q&D, ugly and it takes up unnecessary resources, both for my own service and for Facebook, but it has been tested to work. One known drawback – if the same Facebook user mentions our search term more than once, only the last one is returned.

First – perform the search as always – in simple PHP (make sure to add your own validations):

<?php
$searchterm = 'telia';
$geturl = "http://graph.facebook.com/search?q=" . urlencode($searchterm) . "&type=post";
$fb_reply = file_get_contents($geturl, 0, null, null);
$fb_arr = json_decode($fb_reply, TRUE);
$reply_data = $fb_arr['data'];

Now $reply_data contains an array of posts – something like:

Array
(
 [0] => Array
 (
   [id] => 10002333417237_486897434763199
   [from] => Array
   (
     [name] => some name
     [id] => 100027398117237
   )
  [message] => Missing home
...

Look at the [from]-section to figure out who posted the message. Actually you are going to need all the from->id’s from all posts in the reply from Facebook. Why? Because you want to extract the senders locale and filter out any posts that does not match the locale you require. I use something like this to create an array of Facebook uid’s:

$users = array();
foreach ($reply_data as $pkey => $post) {
 $uid = $post['from']['id'];
 $users[$uid] = $pkey;
}

Next we ask Facebook about the users using FQL. Actually we only want to know about users if they have our locale:

$FQL = sprintf("SELECT uid, name, locale FROM user WHERE uid IN (%s) AND locale = 'da_DK';", implode(array_keys($users), ','));
$fql_query = "http://graph.facebook.com/fql?q=" . urlencode($FQL);
$fql_result = file_get_contents($fql_query, 0, null, null);
$fql_users = json_decode($fql_result, TRUE);

Now $users is a one-to-one mapping from users to matching posts, we’ll “join” the filtered user list with the $users-array and have – a list of posts from users with the requested locale:

$valid_posts = array();
foreach($fql_users['data'] as $dk_user) {
 $uid = $dk_user['uid'];
 $postkey = $users[$uid];
 $valid_posts[] = $reply_data[$postkey];
}

The $valid_posts-array now contains posts matching the needed locale – somewhat like before December 5th 2012.

Conclusion

I wonder why Facebook took the decision they did. It is just asking for workarounds like this. Now instead of one Facebook Graph request I will make two AND Facebook gets to send me a lot more data than I need. The data is still public (for now anyway) – let’s see if it stays that way.