Archive for the ‘The Web’ Category

How to download your Facebook photos

Wednesday, October 27th, 2010

Facebook recently released a feature that allows you to download a static version of your Facebook profile which includes all your videos, photos, your wall and a few bits of periphery information. If you’re planning on quitting Facebook, this is great because it means you don’t lose anything and it’s also a nice offline backup of your Facebook info.

To do this, go to Account -> Account Settings and pick “Download Your Information”.

What this doesn’t get you, however is all the photos you’re tagged in. For this you need to go through the Facebook API. Here’s how to grab all those photos in the best possible quality in a bit of a hacky way. No idea if it violates Facebook’s terms of service – who the hell knows what they are any more?

First of all you’ll need a system which has wget, grep, sed and awk. You can get these for Windows and this should all work, but I’m going to assume you’re logged into something with a bash prompt.

  1. Find your Facebook ID. If like most people you have an alias for your homepage, click on your profile picture and you’ll see “id=xxxxxxxx”. That’s your Facebook ID
  2. Create a URL for calling photos.get in the API. Go to the API documentation page for photos.get and in the Test Console at the bottom, enter your Facebook ID for “subj_id”.
  3. Run the call. Click “Call Method” – you will then see a bunch of code on the right and you will have a URL at the top starting with Click on it to open that page in a new window.
  4. Download the data. Either copy the file from the browser window (ctrl+a, ctrl+c, paste into a file) or run wget -O photos.json "<your URL>" to save your data in a file called photos.json
  5. Grab the images. You can now run cat photos.json | sed 's/,/\n/g' | grep src_big | grep http | sed 's/\\//g' | awk -F\" '{print $4}' | wget -i -. This will pluck out the URL of each photo you’re tagged in and download them to the current folder.

You’ll then see wget going crazy downloading all those photos and when it’s done you’ll have a copy of every photo you’ve been tagged in. After that you’re free to stick them into an album, edit them into your existing offline Facebook profile – whatever you fancy.

OAuth PHP Twitter Class

Tuesday, June 22nd, 2010

After posting my temporary twitter unfollow script the other week I was reminded that Twitter is supposed to be dumping basic auth (sending user name and password in each request to Twitter) this month in favour of the more secure, widely adopted Open Authentication. It turned out Twitter has postponed what they have dubbed the “oauthcalypse” due to the world cup but if you haven’t already, now is the time to get rid of your neat little one-line curl scripts using basic auth (a shame I know but it’s for the best) and find an OAuth alternative in the language of your choice.

I have created a simple TwitterClient class (available under SVN) based on my previous set of basic auth and curl-using functions from a couple of years back. I have left it with my app key in their on purpose, so tweets from it will appear to come from “Strawp’s PHP lib”. Register your own app if you want and use those instead, of course. On the first use of a method you will be prompted to authenticate the class against your twitter login. The secret keys will then be stored in your home folder.

Example usage:

include( "twitter.class.php" );
$tw = new TwitterClient();
$tw->tweet( "I am using oAuth now!" );

The unfollow script has also been updated to use this class.

I finally *get* Yahoo Pipes

Thursday, October 30th, 2008
Yahoo Pipes screenshot

Creating pipes is a bit of a mental leap from traditional programming.

I noticed Yahoo Pipes pretty soon after it launched nearly two years ago but other than aggregating RSS feeds I couldn’t really get into it. The visual psuedo-flow-diagram programming didn’t gel with me at all and I just found myself thinking I could have more easily achieved the same results in less time in PHP.

Well, after chatting with the Yahoo Developers booth at FOWA the other week and this week failing to find a decent UK Xbox 360 release calendar in iCal format I decided to have another crack using the messy data on Gamestation’s website as a source. The result: Gamestation XBox 360 releases. I’ve also created a conversion script in PHP to make that data iCal compatible and therefore importable into pretty much any calendaring system you’d care to mention.

If you’re used to writing screen scraping software in a real programming language there are a few mental barriers you have to clear in order to get anywhere:

  1. There is no regexp “match” function. You have to use substitute if you want to extract a term and make sure you “.*” at either end of it to remove unwanted text.
  2. You have to program loops one after the other instead of nesting logic all inside one big loop. Makes it a very odd thing to read.
  3. Multiline regexps are a headache to write because you have a single-line regexp input. I got around this by making a multi-line string single line with “s/[\r\n]/” before doing proper regexp operations
  4. It’s really quite buggy. Under Firefox in windows drag and drop operations would stop working after a while and under Firefox in Ubuntu after a while the whole edit area would turn into a bit of a mess. Save frequently.

Anyway, it’s pretty good once you get your head around it and it outsources some of the heavy lifting involved in screen-scraping to Yahoo instead of your own site. I’d like to see the ability to export in iCal format as standard in future updates.

The Future of Web Apps Conference 2008

Sunday, October 12th, 2008

A few notes on my first Future of Web Apps (FOWA) conference in the ExCeL centre in London.


Everyone I met was very cool and friendly, from small startups just trying to get noticed to relative tech celebraties and the larger stands like the Microsoft one – everyone came across as genuinely interested in the kind of conversations that make most of my friends eyes glaze over. I was even happily talking about XBox Media Center versus Windows Media Center and why the Silverlight IDE sucks (but the runtime is very good) with the MS guys.

My God do we all look the same though – white middle class guys with short gelled hair wearing jeans and ironic T-shirts. I was finding at times I honestly couldn’t remember whether I’d spoken to someone or not.

Out of the list of speakers, I only really spoke to Blaine Cook (ex lead architect for Twitter, current architect for Fire Eagle) who I found myself sat next to for Diggnation Live. He was a really interesting and friendly guy.


  • Reading between the lines, the reason twitter IM isn’t coming back is because it’s Blaine’s area (he did a talk on Jabber) and not really a priority for the other members of the Twitter team. In terms of bandwidth and resource use Jabber should actually provide a lower overhead than traditional polling over HTTP of the Twitter service and could even allow much of the twitter service to be decentralised to other companies – e.g. to provide SMS gateway functionality again.
  • In about 3 years Facebook should be very open. APIs for accessing/exporting your own friends to other services are on the horizon but not currently a priority.
  • Mark Zuckerberg still delves into the Facebook source code to do bug fixes, despite being head of a staff of 700 employees.
  • Despite being $1.5bn, Mark Zuckerberg still wears North Face fleeces. Who knows why.

Sites of note and people I met

  • Josh from iPlatform – a technology for embedding existing apps and communities inside Facebook as native apps (pitched his business to Jason Calacanis and about 1500 people during the Dragon’s Den section – clearly has sizeable gonads)
  • Zuz from Huddle – an online collaboration tool for businesses
  • The guys from Soocial (pronounced “Social” – they’re Dutch). Soocial is a cloud computing app for your contacts. It takes your contacts from your phone, Gmail, Outlook or your Mac address book and sticks them all in one syncable and backed up place. I’ve signed up to this and it needs work but it has a lot of promise.
  • The guys from Everycity (managed hosting) who I drunkenly found the last tube back with on Thursday.
  • Some guy from GUIMagnets (whose business card doesn’t have a name on it) which is a selection of magnets that look like GUI elements for prototyping apps.
  • Snaffleup, which is like Freecycle but more organised
  • Mapstraction – an abstraction layer for creating embedded map applications with any of the popular mapping services
  • GeoCommons – a resource and tool for finding location specific data and creating interactive maps with it
  • Rummble – a mobile discovery service. If this takes off over the next few years with the uptake of phones like the iPhone and Android based handsets it will be very cool.
  • Last but not least, TactileCRM which is an excellent, uncomplicated CRM system for businesses

There were countless other people I talked to as well who I didn’t get a business card or promotional item off, so I can’t remember their vowel-free unpronouncable domain names, sadly. It was really interesting just seeing what people are building.

The talks are being uploaded to the Carsonified events site and the hilarious episode of Diggnation Live will be up this Friday (17th).