How to download your Facebook photos

October 27th, 2010 by Strawp

Facebook recently released a feature that allows you to download a static version of your Facebook profile which includes all your videos, photos, your wall and a few bits of periphery information. If you’re planning on quitting Facebook, this is great because it means you don’t lose anything and it’s also a nice offline backup of your Facebook info.

To do this, go to Account -> Account Settings and pick “Download Your Information”.

What this doesn’t get you, however is all the photos you’re tagged in. For this you need to go through the Facebook API. Here’s how to grab all those photos in the best possible quality in a bit of a hacky way. No idea if it violates Facebook’s terms of service – who the hell knows what they are any more?

First of all you’ll need a system which has wget, grep, sed and awk. You can get these for Windows and this should all work, but I’m going to assume you’re logged into something with a bash prompt.

  1. Find your Facebook ID. If like most people you have an alias for your homepage, click on your profile picture and you’ll see “id=xxxxxxxx”. That’s your Facebook ID
  2. Create a URL for calling photos.get in the API. Go to the API documentation page for photos.get and in the Test Console at the bottom, enter your Facebook ID for “subj_id”.
  3. Run the call. Click “Call Method” – you will then see a bunch of code on the right and you will have a URL at the top starting with Click on it to open that page in a new window.
  4. Download the data. Either copy the file from the browser window (ctrl+a, ctrl+c, paste into a file) or run wget -O photos.json "<your URL>" to save your data in a file called photos.json
  5. Grab the images. You can now run cat photos.json | sed 's/,/\n/g' | grep src_big | grep http | sed 's/\\//g' | awk -F\" '{print $4}' | wget -i -. This will pluck out the URL of each photo you’re tagged in and download them to the current folder.

You’ll then see wget going crazy downloading all those photos and when it’s done you’ll have a copy of every photo you’ve been tagged in. After that you’re free to stick them into an album, edit them into your existing offline Facebook profile – whatever you fancy.

Recipe for a decent bash shell in Android

August 11th, 2010 by Strawp
  1. Root your phone if you haven’t already and install the latest stable CyanogenMod, which includes bash. If this step sounds scary, stop reading now 🙂
  2. Install ConnectBot:
    Connectbot Market QR code
  3. ConnectBot is a fantastic SSH client, but it also has a local terminal emulator. Open it up and select “local” from the drop down list and give it any nickname.
  4. Open it up to see the local shell works. It will, it’s not bash though and it won’t have tab completion. Hit the menu button and disconnect.
  5. Long-press the connection and select Edit host.
  6. Find the “Post-login automation” option.
  7. enter:
    export PS1="\w\$ "
    cd /

    This step automatically switches you into root mode (you will be prompted by the superuser manager the first time you do this), starts bash, sets your prompt to the working directory and then changes directory to root.

  8. Hit “OK” and then the back button to return to the connections screen. Select the connection to test if it’s all working.
  9. For quick access, you can add a shortcut to the home screen. Long press on the home screen, select “Shortcuts”, “ConnectBot” then the name of your connection.

Tada! A one-click root terminal in bash with tab-completion (courtesy of ConnectBot’s keyboard shortcuts). Much better than the Terminal app every other blog tells you to install 😉

OAuth PHP Twitter Class

June 22nd, 2010 by Strawp

After posting my temporary twitter unfollow script the other week I was reminded that Twitter is supposed to be dumping basic auth (sending user name and password in each request to Twitter) this month in favour of the more secure, widely adopted Open Authentication. It turned out Twitter has postponed what they have dubbed the “oauthcalypse” due to the world cup but if you haven’t already, now is the time to get rid of your neat little one-line curl scripts using basic auth (a shame I know but it’s for the best) and find an OAuth alternative in the language of your choice.

I have created a simple TwitterClient class (available under SVN) based on my previous set of basic auth and curl-using functions from a couple of years back. I have left it with my app key in their on purpose, so tweets from it will appear to come from “Strawp’s PHP lib”. Register your own app if you want and use those instead, of course. On the first use of a method you will be prompted to authenticate the class against your twitter login. The secret keys will then be stored in your home folder.

Example usage:

include( "twitter.class.php" );
$tw = new TwitterClient();
$tw->tweet( "I am using oAuth now!" );

The unfollow script has also been updated to use this class.

Temporary Twitter unfollow script

May 27th, 2010 by Strawp

Update: This now uses the oAuth Twitter class, not the basic auth function library.

Do you ever get people you follow on Twitter who, for some reason have suddenly become really spammy? Normally they’re great, but maybe one evening they’ve come back drunk and are vomiting tweets all over their feed, or maybe they’re live-tweeting some inane reality TV show for the opening night or are taking part in an incredibly uninteresting meme hashtag.

Whatever the reason, you don’t want to ditch them entirely, just until they’ve stopped blabbing on about stuff you don’t care about.

So I made a command line script to get around this. If you want to temporarily unfollow someone, write:

unfollow mrspammy for 1 week

or for several users:

unfollow mrspammy,mrsspammy,mrspammystennispartner for 4 days

Due to the genius of of PHP’s strtodate(), the words after “for” can be pretty much any time period phrase that works with the “+” modifier from the current date. You can also specify exact dates:

unfollow mrspammy until next tuesday


unfollow mrspammy until 12 july

This then looks at users that were previously unfollowed, checks if you want to re-follow them by now and does so if required. Run it without specifying any unfollows to just get this behaviour. This means you can also put unfollow in cron (or whatever) to run periodically and it will automatically check if users need re-following by now.

This is written for Ubuntu. Any Linux will be OK, Mac too probably if you have the paths set up properly. You might even get it working under Windows.

To get it working, you need

  1. My Twitter library under svn
  2. The unfollow script
  3. PHP installed with libcurl

Put the PHP files in the same folder. You then need to:

  1. Edit $confpath in unfollow
  2. Create twitter.conf.php, containing two define() statements with TWITTER_EMAIL and TWITTER_PASSWORD in. These should contain your username and password.
  3. chmod u+x unfollow
  4. optional: Add unfollow to crontab to run at whatever interval you want
  5. optional: make this folder part of path, symlink to unfollow or in some way make unfollow callable without defining the entire path

And you’re good to go. Easy 😉

Let me know if you end up using this or my twitter library at all.

Geeking out with diagrams in ASCII

November 5th, 2009 by Strawp

Last week I discovered ditaa and I can’t quite leave it alone. If you don’t understand why creating flow charts in plain ASCII is awesome, stop reading now.

Basically, you can take something like the Hey Jude flow chart, marked up in ASCII and convert it into a proper version. If your face is doing this: O_o, seriously stop reading now.

I marked that last one up in JavE, which is fine for ASCII art but a bit laborious if you want to build a diagram. On the ditaa home page there’s a link to a screencast of the perl app, Asciio (App::Asciio on CPAN) which is almost perfect. With a couple of tweaks to the /setup/stencils/asciio file, changing all the corners to the ‘+’ character and then another tweak to setup/actions/ to automatically create flat txt and native asciio versions of the diagram, then calling ditaa in the background on the txt file to build a neat PNG version.

I now have a Visio-lite system that can create data flow diagrams suitable for both source code headers and also pasting into more formal documents. So… yeah. How was your week.

If anyone else is this nuts, let me know in the comments.

Streaming media from Linux to a games console

February 24th, 2009 by Strawp

I’ve been using a modified original XBox to stream media from around the house since about 2003, however that same box is getting a little long in the tooth – high res videos with a lot of visual noise chug and stutter and even at the best of times the XBox isn’t doing justice to the higher resolution plasma we now have.

So what are our other options? We’ve got a Wii and two XBox 360s. If someone ends up releasing a game on the PS3 we might even get one of those as well.


This is doable but completely pointless compared to the original xbox. I’d need to transcode everything in to the lower-quality sorenson flash video codec (whatever format it was in, this will be lower quality) and it won’t output at an HD resolution.


I’m told this works very well as a media playing client and gets along much better than the MS products with other machines on the network (as you’d expect). I don’t have one though.

XBox 360

There are basically two options for streaming to an XBox 360: Setting up Windows Media Center on your Windows-based machine or creating a UPnP (Universal Plug ‘n’ Play) based media server on the network. This is Linux (Ubuntu 8.10) we’re talking about so option one is right out. There are a few UPnP servers for Linux around though. Here’s a breakdown:

  • MediaTomb (package name, mediatomb)

    This is a very slick service which is easy to set up, automatically scans your media and presents it in an organised manner. It supports video, pictures and audio with a variety of formats and has the ability to transcode between formats on the fly using VLC. Very cool. Works with everything except a 360.

  • uShare (package name ushare)

    This is pretty cut-down. The video playback works fine but there is no rescan option at the time of writing meaning you have to restart the service for new videos to appear. Not good if you want to watch something the minute it arrives on the server. This can be partly alleviated by creating a cron job to restart the server daily, but it’s not ideal. The audio library on uShare is also completely useless. It seems to attempt to organise everything, completely fails and only lets you see a handful of tracks in amongst folders which don’t even contain audio. It would have been better if it had just let you walk through the folders like in video mode.

  • X360MediaServe (no deb package)

    This requires an old-fashioned manual setup as it’s not in any Ubuntu repository but works well. However it ONLY supports audio so is not suitable for my needs.

  • FUPPES (no deb package)

    Steer clear of this if you don’t like the idea of not only an old fashioned manual configure, make, make install style setup but having to google error messages in order to even get it visible from an XBox. It’s almost even worth just using uShare and x360MediaServe both at the same time so that at least you have reasonably hassle-free video and audio.

    If you do happen to have enough patience and you’ve managed to craft the config files into something workable it turns out that this is actually one of the better least rubbish solutions for streaming to 360. The video works and although you have to hammer on the vfolder.cfg file (which doesn’t get built for you) in order to get something workable going it does kind of do the job in a very basic and raw way. Given a bit more time and a decent setup script this might turn out to be quite decent server, but then given a rescan option and proper music support in uShare I wouldn’t even be trying this out.

  • TwonkyMedia Server

    Twonky sticks out in this list because although there is a native linux server available it’s locked to free trial only before you have to part with your readies. This proprietary nature of software releases is so foreign to Linux it left me reeling in confusion and I gave up on it before even getting it working. If none of that bothers you then it’s probably the best out of the entire list. I’m a man of principles though. Or a tight-arse.

  • Samba

    Inexplicably notable by its absence is Samba, SMB, straight-forward common Windows file sharing. Every OS on earth understands it and I’d been sharing media using it for years already so why doesn’t the 360 support it? No DRM control? Is it because the protocol has been tainted by FOSS under the evil hand of the EU courts? Whatever the reason, it’s pretty typical Microsoft.

At the time of writing I have FUPPES serving both video and music although I can’t map the 360 media player’s “Artist” and “Album” drill down options to it. After a lot of fiddling I got most video working with a drill-down directory list. I haven’t given up on FUPPES yet but the truth is that despite the lack of rescan and decent music support, uShare is still the better option for video streaming at the moment. Install via apt, edit a couple of lines in the config file and then that’s the whole thing set up. Sure you have to restart it to recache the media list but it’s still less effort than FUPPES is.


If you want to stream music, use x360MediaServe. If you want to stream video, use uShare. If you have a ton of patience or you’re a masochist, use FUPPES. If you want to stream both video and music and you have better things to do, use MediaTomb and get a PS3, or you could try both x360MediaServe and uShare at the same time. Bit messy though, right?

Oh hey – could this be the PS3’s first worthwhile exclusive? 😉

Update: Experiment1106 on Twitter pointed out PS3MediaServer which is apparently the best solution for streaming to PS3s (and claims basic 360 support).

iPlayer download class in Java

January 22nd, 2009 by Strawp

I just got a G1 (A.K.A. The GooglePhone) the other week so I’m trying to pick up Java again so I can write some apps for it. I’m bloody terrible at Java and I’ve had flu for the last week but nevertheless I’ve managed to hack together a class which can download a TV show off iPlayer given the programme ID (PID).

It’s not totally finished yet – I’ve got a warning about not releasing connections there that I need to clear up (answers in the comments please!). You can take a look at the code on my SVN repo:

Once this is all stable I’ll be having a go at making a GUI and getting it onto my phone.

Update: All working error-free now! GUI next…

New Facebook to GCal Greasemonkey script

December 11th, 2008 by Strawp

I’ve been using the original Facebook to Google Calendar script since it was first launched but for whatever reason the redesigns of Facebook seemed to run away from the original author and it stopped working properly so I decided to branch my own version.

v1.0.0 is largely untested and in “works for me” stage so try it out and give feedback!

Note that you may need to refresh the event page in order for the script to kick in because of how Facebook now loads pages in using AJAX.

The BNP Hate Factor League

November 26th, 2008 by Strawp

As I’m sure you all noticed last week, the latest fun leaked data from a polical party came from the BNP who somehow had their entire membership list leaked onto the internet. For people like myself this presented two fun opportunities:

  1. To do a little bit of geographical and statistical analysis on some odd data
  2. To laugh at a bunch of hate-filled racists

There were quite a few nuggets of statistical analysis in the first couple of days: A proximity checker to see which of your neighbours members, the obligatory google maps mashups (since, sensibly taken down), a sort of heat map and the Grauniad did an excellent map broken into electoral wards. They were all pretty good, but they still rather suffered from the problem that you see “hot spots” in areas which are naturally population nodes. There was no accounting for population density.

Anyway, in a spare moment I took a copy of the database, cleaned up the postcode information a bit, ran it through a geocoder to get lat and long data and then ran that through a lookup for population density and then grouped the data by postcode area. What I now have is a count of each person in a postcode area, divided by the population density – this should then give a population normalised rank of how hate-filled post code areas are. Anyway, here’s the Top 40, Top Of The Pops style:

Rank Postal Area Town County Members Population Density Hate Factor
1 LE67 Coalville Leicestershire 56 0.621 90.125
2 S63 Bolton-on-Dearne Rotherham 50 0.557 89.846
3 HX3 Boothtown Halifax 35 0.563 62.182
4 LS27 Morley Leeds 46 0.772 59.618
5 BD22 Oakworth Keighley 32 0.598 53.556
6 CR4 263-265 London Road Mitcham 24 0.48 49.96
7 WF2 Wakefield West Yorkshire 27 0.543 49.727
8 HX2 Illingworth Halifax 27 0.551 48.974
9 NG10 Long Eaton Nottingham 27 0.551 48.97
10 DE55 Alfreton Derbyshire 35 0.735 47.594
11 S75 Silkstone Common Barnsley 33 0.701 47.047
12 LS15 Crossgates Leeds 35 0.772 45.362
13 CW7 Winsford Cheshire 9 0.199 45.124
14 M27 Swinton Manchester 9 0.199 45.124
15 BH1 Bournemouth Dorset 7 0.158 44.439
16 S70 Kendray Barnsley 20 0.463 43.202
17 BD13 Queensbury Bradford 24 0.585 41.025
18 L26 Halewood Knowsley 9 0.238 37.871
19 E4 Chingford Hatch London 12 0.32 37.47
20 B37 Solihull West Midlands 27 0.743 36.339
21 S6 Riverlin Sheffield 19 0.528 35.99
22 DE24 Stenson Fields Derby 26 0.772 33.697
23 N18 Aberdeen Road London 8 0.24 33.307
24 B63 Halesowen West Midlands 13 0.414 31.382
25 S71 Carlton Barnsley 19 0.609 31.192
26 NE34 South Shields Tyne & Wear 24 0.772 31.105
27 CV6 Bell Green Coventry 24 0.772 31.105
28 WF3 Tingley Wakefield 24 0.772 31.105
29 HD7 Leymoor, Golcar Huddersfield 17 0.551 30.835
30 WS9 Aldridge Walsall 17 0.551 30.835
31 CH2 Mickle Trafford Chester 8 0.267 29.923
32 DE21 Oakwood Derby 23 0.772 29.809
33 BD20 Glusburn Keighley 20 0.68 29.405
34 NG17 Kirkby-in-Ashfield Nottinghamshire 31 1.066 29.072
35 DE15 Burton-on-Trent Staffordshire 22 0.772 28.513
36 S5 Sheffield South Yorkshire 18 0.643 27.995
37 S12 Sheffield South Yorkshire 8 0.289 27.649
38 B14 Kings Heath Birmingham 14 0.512 27.347
39 HD3 Longwood Huddersfield 17 0.623 27.303
40 LS9 Leeds West Yorkshire 21 0.772 27.217

I finally *get* Yahoo Pipes

October 30th, 2008 by Strawp
Yahoo Pipes screenshot

Creating pipes is a bit of a mental leap from traditional programming.

I noticed Yahoo Pipes pretty soon after it launched nearly two years ago but other than aggregating RSS feeds I couldn’t really get into it. The visual psuedo-flow-diagram programming didn’t gel with me at all and I just found myself thinking I could have more easily achieved the same results in less time in PHP.

Well, after chatting with the Yahoo Developers booth at FOWA the other week and this week failing to find a decent UK Xbox 360 release calendar in iCal format I decided to have another crack using the messy data on Gamestation’s website as a source. The result: Gamestation XBox 360 releases. I’ve also created a conversion script in PHP to make that data iCal compatible and therefore importable into pretty much any calendaring system you’d care to mention.

If you’re used to writing screen scraping software in a real programming language there are a few mental barriers you have to clear in order to get anywhere:

  1. There is no regexp “match” function. You have to use substitute if you want to extract a term and make sure you “.*” at either end of it to remove unwanted text.
  2. You have to program loops one after the other instead of nesting logic all inside one big loop. Makes it a very odd thing to read.
  3. Multiline regexps are a headache to write because you have a single-line regexp input. I got around this by making a multi-line string single line with “s/[\r\n]/” before doing proper regexp operations
  4. It’s really quite buggy. Under Firefox in windows drag and drop operations would stop working after a while and under Firefox in Ubuntu after a while the whole edit area would turn into a bit of a mess. Save frequently.

Anyway, it’s pretty good once you get your head around it and it outsources some of the heavy lifting involved in screen-scraping to Yahoo instead of your own site. I’d like to see the ability to export in iCal format as standard in future updates.