Twitter OAuth with PHP and cURL multi

No Gravatar

I am maintaining a website that aggregate a large number of Twitter accounts’ tweets. For this purpose we’ve been given one of the accounts with the high API request rate limit – usually it is possible to call the Twitter API’s 150 times every hour. We have a rate limit of 20.000 hourly requests and we use most of them to keep up a near real time picture of the activity of the user base.

Recently Twitter disabled regular basic HTTP authentication supporting only OAuth – which generally is a good thing, In this case we simply read the RSS-version of the user_timeline and OAuth seems overkill for reading a simple public time line.¬†Aaaanyways – we are using PHP with cURL multi and OAuth introduces a bit of extra code. Sure – there are plenty of OAuth API’s and OAuth API’s for Twitter out there, but the specific combination of PHP cURL multi and GET’ing user_timelines required a combination of Google’ing and a bit of guesswork.

Preparations

– First¬†Register your application with Twitter – You’ll need a couple of tokens and secrets from your app-page and from the specific token page.
– If you need to make more than 150 requests per hour, apply for¬†whitelisting here. ¬†It’ll take a while for Twitter to process your request.
– Make sure you have cURL for PHP installed

Specifically preparing the GET connections for cURL presented a challenge – this piece of code did the trick for us:

<?php
// $url_array is an array of Twitter RSS-feed URL's that we need to read
$mh = curl_multi_init();
foreach($url_array as $i => $item) {
	// Keys and secrets from http://dev.twitter.com/apps/
	$consumer_secret = '<get your own from http://dev.twitter.com/apps/_your_app_id_/>';
	$token_secret = '<get your own from http://dev.twitter.com/apps/_your_app_id_/my_token>';

	// Build ud an array of OAuth parameters
	$params = Array();
	$params['oauth_consumer_key'] = '<get your own from http://dev.twitter.com/apps/_your_app_id_/>';
	$params['oauth_token'] = '<get your own from http://dev.twitter.com/apps/_your_app_id_/my_token>';
	$params['oauth_signature_method'] = 'HMAC-SHA1';
	$thetime = time();
	$params['oauth_timestamp'] = $thetime;
	$params['oauth_nonce'] = SHA1($thetime);
	$params['oauth_version'] = '1.0';

	// Sort the array alphabetically
	ksort($params);

	// Build the paramter string for the GET request
	$concatenatedParams = '';
	foreach($params as $k => $v)
	{
	  $k = urlencode($k);
	  $v = urlencode($v);
	  $concatenatedParams .= "{$k}={$v}&";
	}
	$unencodedParams = substr($concatenatedParams,0,-1);
	// URL-encode the parameters for signature use
	$concatenatedParams = urlencode(substr($concatenatedParams,0,-1));
	$signatureBaseString = "GET&".urlencode($item['url'])."&".$concatenatedParams;

	$params['oauth_signature'] = base64_encode( hash_hmac('sha1', $signatureBaseString, $consumer_secret."&".$token_secret, true) );

	// Initiate a new cURL connection and set up it's URL
	$conn[$i] = curl_init();
	curl_setopt($conn[$i], CURLOPT_URL, $item['url'] . '?' . $unencodedParams);
	curl_setopt($conn[$i], CURLINFO_HEADER_OUT, 1);

	// Build the HTTP header for the request
	$curlheader = Array();
	$curlheader[]='Content-Type: application/x-www-form-urlencoded';
	$curlheader[] = 'Authorization: OAuth';
	foreach($params as $ky=>$va) {
		$curlheader[] = "$ky=$va,\n";
	}

	// Initiate a new cURL connection and assign URL + HTTP header to it
	$conn[$i] = curl_init($item['url']);
	curl_setopt($conn[$i], CURLOPT_HTTPHEADER, $curlheader);
	curl_setopt($conn[$i], CURLOPT_HTTPGET, 1);

	// Add the connection to the cURL multi handle
	curl_multi_add_handle ($mh,$conn[$i]);
}

// Now execute curl_multi_exec on the $mh handle
This can no doubt be improved and secured in many ways, but since this specific example runs from as a cron-job/a PHP CLI Рnot on a server that replies to  inbound connections Рthe keys are not secured any further in this implementation.

Will ideas of the Massachusetts law spread?

No Gravatar

On reddit, I found this article regarding a new law i Massachusetts that is to increase the security of personally identifiable information.

If you have personally identifiable information (PII) about a Massachusetts resident, such as a first and last name, then you have to encrypt that data on the wire and as it’s persisted. Sending PII over HTTP instead of HTTPS? That’s a big no no. Storing the name of a customer in SQL Server without the data being encrypted?  No way, Jose. You’ll get a fine of $5,000 per breach or lost record. If you have a database that contains 1,000 names of Massachusetts residents and lose it without the data being encrypted that’s $5,000,000. Yikes.

The idea is good (yay – better security for anyone registering on a website) and bad (expensive). As a consultant in a country not covered by the law I could care less, but is this in general a good idea? Securing data and sending sensitive information across secure connections is always a good idea, and it could hit the EU soon. Documenting how you secure the Personally Identifiable Information (PII)? Who can argue with that being a good idea? As an IT consultant you won’t hear me cry myself to sleep.

But the proportions of the legislation seem unclear. If you log on to my website (securely of course), and I feed you the following pages through clear text HTTP – can I be fined if your name appear on the page as in “Reader XYZ is logged in”?

I guess so.

It strikes me that there is no level of sensitivity defined – anything considered personal must be secured. As a legislator, it seems very easy to do this when you don’t have to pick up the bill.

If this kind of legislation should hit Europe, I hope someone would elaborate a bit on the do’s and dont’s:

  • Are all kinds of data included? Can I be fined if you, as a Massachusetts citizen, post a comment to this and include your name? (the database is secured, but nothing is sent through HTTPS)
  • Would it make sense to allow users to wave their rights, and thereby allow users to work with online applications that are not 100% secure?

Javascript performance on current browsers

No Gravatar

Found this little test which gives an indicator as to Javascript performance in your browser.

On my system (Windows XP on two year old Lenovo T60p laptop) i tried to run it ten times on my browsers with all plugins disabled (lower is better):

Google Chrome: 297,7
Firefox 3.5 beta 4: 340
Firefox 3.0.10: 408,9
Internet Explorer 8: 631,3

As the score is time, lower is better. This is interesing because sites uses Javascript more and more, and as we work more and more online with more applications in the cloud, the Javascript engine has a lot to say about our perception of overall performance.

I got a bit disappointed about my Atom-based netbook – specifically Ubuntu 9.04 on that machine. It never went below 1800 (Firefox 3.0.10) and on the same hardware, the Windows-browsers gives me minimums of 1500 and 2800 for Firefox 3.0.10 and IE8 respectively. Gotta find some tweaks there.

Netbooks – the necessary new design test-tool

No Gravatar

Writing this on my Lenovo S10e netbook, I am furious. It’s before mid-day and yet I have had two experiences of software designs that did not consider netbooks a platform – or at least the new low screen resolution these computers imply. And before you call me a whiner (besides the fact that you’d be somewhat right), this is just a description of the changes I will make to include netbook users as an audience of software and website in the future.

Whine #1: Twice I’ve been cripled by software that saw the low screen resolution as a handicap – one of them on purpose. First I installed Pidgin – the cool cross-platform/cross-protocol IM and IRC client. I like it a lot, but on Windows some dialogs are too big and will not allow me to navigate to the OK/Cancel buttons at the bottomof the settings dialog. Fortunately this is Open Source stuff, so I can just participate and actively fix this myself.

Whine #2: I had to install a printer driver for my HP Photosmart 2575 printer – the install took over one hour because of some “ingenious” package system. That obviously poor user experience decision aside – the minimum requirement for the printer driver is a screen resolution of 800×600 pixels. My S10e runs at 1024 x 576. The consequence – I cannot print from my netbook in Windows because the printer driver won’t finish installing, as it has an irrelevant requirement. Fortunately I am dual-booting with Ubuntu, which has excellent support for my printer (without the requirement)

I’ve read somewhere that 20% of all computers that will be sold in 2009 will be netbooks. Some producers (including Asus) will stop production of 8.9″ -screen netbooks. The 10-inchers seems to dominate right now, and probably for the rest of the year (note: my guess only). Every company designing software will have to take this into account before they ship the next version of any product with a user interface.

I’ve worked on so many web projects with art directors saying: “Nobody runs 640×480 or 800×600 anymore”. Hey we know – but do you know how many users run with their browser maximized because a designer thought up a design that required it? Web designers don’t own the real estate of the users screen resolution – they can only hope to own the area the browser is sized to – and you have to count on users having at least one open history/bookmarks sidebar and/or plug-in and/or Google/MSN/Web developer toolbar enabled.

My point should be rather obvious: It is vital to include netbooks as testplatforms for serious UI-designers or your product will no longer be compatible with the equipment of the customers you want. The rules have changed – live with it.

You may argue that netbook users are asking for it, but if the competing website or software support the netbooks and you don’t – the users is no longer making the decision of leaving you.