Strawp.net

Recently, an Android version of Winamp was released and included in its feature list was very useful wireless syncing with the Windows version of Winamp. Unfortunately, my file server at home doesn’t run Windows and Winamp under Wine is an unstable mess, so attempting to use that feature was out. Like a good Linux user, I didn’t take this lying down – we don’t sit around like Windows users waiting for someone to build the solution for us, we cobble something together using tools already available! What I decided to do was to emulate the mechanism that I came up with when I built my Car PC. Basically, the core of wireless syncing in the Car PC was rsync, combined with a little bit of logic to get a list of music track locations to pull over. By offloading some of that logic onto the server, this seemed possible.

First, I installed an rsync app for Android and tested it out. Rsync backup is just a front end for rsync, which is perfect because it lets you use all the normal rsync options, allows public key authentication (so you never need to enter a password) and displays rsync’s output to a log window. I wanted to pull files from my server so in my rsync profile I selected “rsync in reverse” and added the command line options “-vHrtDL --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --progress --partial” most of which is the default for a new profile, but I added:

• -L, to follow symlinks
• --progress, to display file transfer progress
• --partial, to allow file transfer resuming

Android won’t know any of the new music files have been added yet, so they won’t show up in any music players (which all look up available tracks in the built-in Android media database), so you need to install something like SDrescan to run after the files have copied over which will magically make them visible in the music app of your choice.

That’s the phone side sorted out, I just need something sensible to point it at – I don’t want rsync recursing over my entire music collection wirelessly, it would take ages.

On the server, I’m running the (somewhat ageing) Firefly Media Server, which indexes and serves music for my Roku Soundbridge. I’d previously written a data abstraction class in PHP for the sqlite database it runs on, so I used that to produce a script (to run periodically under cron) which picks a list of albums from a playlist (created by another script) and creates a bunch of symlinks to them in the folder that I set as the target in my rsync profile, the idea being I have a regularly changing folder which one rsync command can look at to pick up new music.

I now have wireless syncing that works anywhere with an internet connection, as well as on the local network (unlike Winamp) and the idea of keeping a static rsync profile and using a source folder full of symlinks provides a cunning way of pushing content onto your phone. It would be quite easy to create something like a web interface to pick files on the server and have symlinks of them created on a single rsync source folder. On top of this, Rsync backup provides hooks for Tasker to link into, so you could set this up to run automatically at some opportune moment.

A couple of months ago Samy Kamkar presented a cool hack at the Black Hat Conference which demonstrated that by using a cunningly constructed URL against specific internet routes, you could inject some javascript on their configuration pages and trick them into sending their own MAC address (the router’s own unique code) to a script which would use it to look up their address on Google’s API, thus telling the attacker a pretty good estimate of your physical location.

This was only possible because as well as photographing everything, the Google Streetview cars have been recording the locations of every wireless access point they encountered. By recording the signal strengths of certain access points by the location they were observed in it’s possible to do a simple triangulation calculation and get a pretty good estimate of where the access point is.

I found this pretty fascinating, so I created a little PHP script to use this trick, perhaps it could be used to help locate a stolen laptop, for example. The script works like this:

1. Attempt to scan using the wireless network adapter for any nearby access points. Save their key details (BSSID, signal strength and most importantly, the MAC)
2. Compile all this into POST data and send it to Google using libcurl. Even without nearby MACs Google’s location API will do a better job at locating the machine than the usual GeoIP services.
3. Get a bunch of data back from Google, including longitude and latitude estimates and a street address. For ease this also gives you a Google Maps link

The accuracy depends on how many neighbours your wireless card picks up and how much data Google has harvested from the wireless networks on your road, but for most people it will be accurate almost to the right house number.

The script is on my SVN server for download

Facebook recently released a feature that allows you to download a static version of your Facebook profile which includes all your videos, photos, your wall and a few bits of periphery information. If you’re planning on quitting Facebook, this is great because it means you don’t lose anything and it’s also a nice offline backup of your Facebook info.

To do this, go to Account -> Account Settings and pick “Download Your Information”.

What this doesn’t get you, however is all the photos you’re tagged in. For this you need to go through the Facebook API. Here’s how to grab all those photos in the best possible quality in a bit of a hacky way. No idea if it violates Facebook’s terms of service – who the hell knows what they are any more?

First of all you’ll need a system which has wget, grep, sed and awk. You can get these for Windows and this should all work, but I’m going to assume you’re logged into something with a bash prompt.

1. Find your Facebook ID. If like most people you have an alias for your homepage, click on your profile picture and you’ll see “id=xxxxxxxx”. That’s your Facebook ID
2. Create a URL for calling photos.get in the API. Go to the API documentation page for photos.get and in the Test Console at the bottom, enter your Facebook ID for “subj_id”.
3. Run the call. Click “Call Method” – you will then see a bunch of code on the right and you will have a URL at the top starting with https://api.facebook.com/method/photos.get?subj_id=. Click on it to open that page in a new window.
4. Download the data. Either copy the file from the browser window (ctrl+a, ctrl+c, paste into a file) or run wget -O photos.json "<your URL>" to save your data in a file called photos.json
5. Grab the images. You can now run cat photos.json | sed 's/,/\n/g' | grep src_big | grep http | sed 's/\\//g' | awk -F\" '{print $4}' | wget -i -. This will pluck out the URL of each photo you’re tagged in and download them to the current folder.

You’ll then see wget going crazy downloading all those photos and when it’s done you’ll have a copy of every photo you’ve been tagged in. After that you’re free to stick them into an album, edit them into your existing offline Facebook profile – whatever you fancy.

1. Root your phone if you haven’t already and install the latest stable CyanogenMod, which includes bash. If this step sounds scary, stop reading now
2. Install ConnectBot:
   
3. ConnectBot is a fantastic SSH client, but it also has a local terminal emulator. Open it up and select “local” from the drop down list and give it any nickname.
4. Open it up to see the local shell works. It will, it’s not bash though and it won’t have tab completion. Hit the menu button and disconnect.
5. Long-press the connection and select Edit host.
6. Find the “Post-login automation” option.
7. enter:

   su
   bash
   export PS1="\w\$ "
   cd /

   This step automatically switches you into root mode (you will be prompted by the superuser manager the first time you do this), starts bash, sets your prompt to the working directory and then changes directory to root.

8. Hit “OK” and then the back button to return to the connections screen. Select the connection to test if it’s all working.
9. For quick access, you can add a shortcut to the home screen. Long press on the home screen, select “Shortcuts”, “ConnectBot” then the name of your connection.

Tada! A one-click root terminal in bash with tab-completion (courtesy of ConnectBot’s keyboard shortcuts). Much better than the Terminal app every other blog tells you to install

After posting my temporary twitter unfollow script the other week I was reminded that Twitter is supposed to be dumping basic auth (sending user name and password in each request to Twitter) this month in favour of the more secure, widely adopted Open Authentication. It turned out Twitter has postponed what they have dubbed the “oauthcalypse” due to the world cup but if you haven’t already, now is the time to get rid of your neat little one-line curl scripts using basic auth (a shame I know but it’s for the best) and find an OAuth alternative in the language of your choice.

I have created a simple TwitterClient class (available under SVN) based on my previous set of basic auth and curl-using functions from a couple of years back. I have left it with my app key in their on purpose, so tweets from it will appear to come from “Strawp’s PHP lib”. Register your own app if you want and use those instead, of course. On the first use of a method you will be prompted to authenticate the class against your twitter login. The secret keys will then be stored in your home folder.

Example usage:

include( "twitter.class.php" );
$tw = new TwitterClient();
$tw->tweet( "I am using oAuth now!" );

The unfollow script has also been updated to use this class.

Update: This now uses the oAuth Twitter class, not the basic auth function library.

Do you ever get people you follow on Twitter who, for some reason have suddenly become really spammy? Normally they’re great, but maybe one evening they’ve come back drunk and are vomiting tweets all over their feed, or maybe they’re live-tweeting some inane reality TV show for the opening night or are taking part in an incredibly uninteresting meme hashtag.

Whatever the reason, you don’t want to ditch them entirely, just until they’ve stopped blabbing on about stuff you don’t care about.

So I made a command line script to get around this. If you want to temporarily unfollow someone, write:

unfollow mrspammy for 1 week

or for several users:

unfollow mrspammy,mrsspammy,mrspammystennispartner for 4 days

Due to the genius of of PHP’s strtodate(), the words after “for” can be pretty much any time period phrase that works with the “+” modifier from the current date. You can also specify exact dates:

unfollow mrspammy until next tuesday

or

unfollow mrspammy until 12 july

This then looks at users that were previously unfollowed, checks if you want to re-follow them by now and does so if required. Run it without specifying any unfollows to just get this behaviour. This means you can also put unfollow in cron (or whatever) to run periodically and it will automatically check if users need re-following by now.

This is written for Ubuntu. Any Linux will be OK, Mac too probably if you have the paths set up properly. You might even get it working under Windows.

To get it working, you need

1. My Twitter library under svn
2. The unfollow script
3. PHP installed with libcurl

Put the PHP files in the same folder. You then need to:

1. Edit $confpath in unfollow
2. Create twitter.conf.php, containing two define() statements with TWITTER_EMAIL and TWITTER_PASSWORD in. These should contain your username and password.
3. chmod u+x unfollow
4. optional: Add unfollow to crontab to run at whatever interval you want
5. optional: make this folder part of path, symlink to unfollow or in some way make unfollow callable without defining the entire path

And you’re good to go. Easy

Let me know if you end up using this or my twitter library at all.

Last week I discovered ditaa and I can’t quite leave it alone. If you don’t understand why creating flow charts in plain ASCII is awesome, stop reading now.

Basically, you can take something like the Hey Jude flow chart, marked up in ASCII and convert it into a proper version. If your face is doing this: O_o, seriously stop reading now.

I marked that last one up in JavE, which is fine for ASCII art but a bit laborious if you want to build a diagram. On the ditaa home page there’s a link to a screencast of the perl app, Asciio (App::Asciio on CPAN) which is almost perfect. With a couple of tweaks to the /setup/stencils/asciio file, changing all the corners to the ‘+’ character and then another tweak to setup/actions/file.pl to automatically create flat txt and native asciio versions of the diagram, then calling ditaa in the background on the txt file to build a neat PNG version.

I now have a Visio-lite system that can create data flow diagrams suitable for both source code headers and also pasting into more formal documents. So… yeah. How was your week.

If anyone else is this nuts, let me know in the comments.

I’ve been using the original Facebook to Google Calendar script since it was first launched but for whatever reason the redesigns of Facebook seemed to run away from the original author and it stopped working properly so I decided to branch my own version.

v1.0.0 is largely untested and in “works for me” stage so try it out and give feedback!

Note that you may need to refresh the event page in order for the script to kick in because of how Facebook now loads pages in using AJAX.

Creating pipes is a bit of a mental leap from traditional programming.

I noticed Yahoo Pipes pretty soon after it launched nearly two years ago but other than aggregating RSS feeds I couldn’t really get into it. The visual psuedo-flow-diagram programming didn’t gel with me at all and I just found myself thinking I could have more easily achieved the same results in less time in PHP.

Well, after chatting with the Yahoo Developers booth at FOWA the other week and this week failing to find a decent UK Xbox 360 release calendar in iCal format I decided to have another crack using the messy data on Gamestation’s website as a source. The result: Gamestation XBox 360 releases. I’ve also created a conversion script in PHP to make that data iCal compatible and therefore importable into pretty much any calendaring system you’d care to mention.

If you’re used to writing screen scraping software in a real programming language there are a few mental barriers you have to clear in order to get anywhere:

1. There is no regexp “match” function. You have to use substitute if you want to extract a term and make sure you “.*” at either end of it to remove unwanted text.
2. You have to program loops one after the other instead of nesting logic all inside one big loop. Makes it a very odd thing to read.
3. Multiline regexps are a headache to write because you have a single-line regexp input. I got around this by making a multi-line string single line with “s/[\r\n]/” before doing proper regexp operations
4. It’s really quite buggy. Under Firefox in windows drag and drop operations would stop working after a while and under Firefox in Ubuntu after a while the whole edit area would turn into a bit of a mess. Save frequently.

Anyway, it’s pretty good once you get your head around it and it outsources some of the heavy lifting involved in screen-scraping to Yahoo instead of your own site. I’d like to see the ability to export in iCal format as standard in future updates.

Just added a new repository to the site: PHP Libraries. Currently in there are working classes or functions for:

• Firefly Media Server – a class for direct access to the songs database
• Roku Client – a client for Roku music players like the Soundbridge. Allows remote scripting etc
• twitter – a bunch of functions for posting to, or getting info from a twitter account
• Delicious – a class for (currently only) getting delicious bookmarks. Will add other methods if/when they’re needed for other scripts.

And there’s a ping.fm class in there that I might work on if I can be bothered to get an App key for it.

Head to svn.strawp.net/lib to browse them, svn co http://svn.strawp.net/lib to check the lot out.

Update: Just added svn.strawp.net/scripts/ which currently just has my twitter command line client in it.