Quick and Dirty DVR Redux

December 19th, 2012 by Strawp

Dropbox is handy
It’s over a year since my blog post entitled “Quick and Dirty DVR” and I’ve been using and tweaking the code ever since. What I have now is a pretty autonomous and functional video recorder based around that core record script.

All code mentioned in this post can be found either in my scripts or lib svn repository. I apologise for the hard-coded paths…

What I’ve added

  1. Scheduling & collision detection
  2. File management
  3. Ad detection
  4. Remote scheduling

Automatic scheduling

The first thing was to get the script to find and record TV shows automatically. To do this, the script needs to know:

  1. A source of TV listings
  2. A list of shows that I watch
  3. The last episode number of a given show that has already been recorded
  4. When all subsequent shows are on, in the future

The first step is easy – just install the XMLTV package and set up a cron tab: 0 2 * * * /usr/bin/tv_grab_uk_rt --quiet > /share/tv/listings.xml. This will check TV listings nightly.

The list of TV shows is simply a flat text file – one show title per line. I took the decision to allow regular expressions in show titles when the show name starts with “like”. For example, QI and QI XL are largely interchangeable so the entry for QI is “like QI( XL)?”, meaning the “XL” part can be there or not but it will be considered to be the same show.

Then I just have to write something to work out which TV shows I already have. It’s very helpful to have your TV neatly organised for this. All my shows are in the path /share/tv/[name of tv show]/[name.of.tv.show].s[series]e[episode].[subtitle].[extension], so inside tv.func.php is the function getLastEpisodeOf($name,$dir) which returns text describing which episodes of that show should be recorded, e.g. “Peep Show after season 8 episode 2″.

Next the listings file needs to be parsed. For a while I had some very dumb parsing which went through each line of the subscriptions file and picked the first available episode of whichever show it was searching for. This was flawed as the scheduler wasn’t aware of other shows being recorded and therefore clashes in scheduling were inevitable. The most I could do was to detect the collisions between shows after the schedule was written and manually correct them.

After a bit of thought I replaced the scheduler with a new script which takes the order in the list of TV shows file as a priority for each show and checks that any new show attempting to be recorded does not clash with a show already scheduled. If this happens, alternative times are checked and used. The presence of “+1″ channels makes this a lot more robust. If a show doesn’t have a suitable recording slot, it’s dropped from the schedule and a warning is produced in the crontab output as a comment.

This is an extract of my subscriptions file:

Fresh Meat
The Killing
Peep Show
Wallander
Bang Goes the Theory
Doctor Who
Sherlock
Top Gear
The Simpsons
like QI( XL)?
like Have I Got.* News For You
The Snowman and The Snowdog
The Royal Institution Christmas Lectures

And this is an extract of how the recording schedule looks for the Christmas 2012 period:


# Clash: The Royal Institution Christmas Lectures s1e1 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Air: The Elixir of Life on BBC Four for 61 at 2012-12-26 20:00-21:01 with EastEnders s1e4556 2012-12-26 Wednesday on BBC One for 31 at 2012-12-26 20:30-21:01
# Clash: Tangled on ITV1+1 (Central) for 111 at 2012-12-25 16:10-18:01 with Doctor Who s7e6 The Snowmen on BBC One for 61 at 2012-12-25 17:15-18:16
# No suitable schedule found for Tangled

# Peep Show after Season 08 episode 04
30 22 23 12 * root /home/iain/bin/record 'Peep Show s8e5 Chairman Mark on Channel 4 for 31' # Jeremy is desperate to avoid living with Super Hans and rekindles his relationship with Mark's sister Sarah, moving in with her and her five-year-old son Joshy. In an attempt to deal with a damp patch on his bedroom wall, Mark launches a campaign to be elected chairman of Apollo House's freehold committee. Starring Robert Webb and David Mitchell. The final episode is tomorrow at 10pm.
00 22 24 12 * root /home/iain/bin/record 'Peep Show s8e6 Quantocking II on Channel 4 for 31' # Dobby is offered a job in New York by ex-boyfriend Simon, but is uncertain whether to accept the offer. She suggests to Mark they should go for a weekend break in the country, while Jeremy decides it is time to get a few things off his chest. Comedy, starring David Mitchell and Robert Webb. Last in the series.

# Doctor Who after Season 07 episode 05
15 17 25 12 * root /home/iain/bin/record 'Doctor Who s7e6 The Snowmen on BBC One for 61' # Matt Smith returns as the Time Lord, who is in mourning after losing Amy and Rory to the Weeping Angels and determined to avoid getting mixed up in the universe's problems. But a call for help whisks him back to Christmas Eve 1892, where a trio of old friends and a plucky young governess called Clara need him to take on a chilly menace that comes with the snowfall. Will he be persuaded to abandon his new life as a recluse and defend his beloved Earth once more? Jenna-Louise Coleman joins the Doctor as his new companion - although whether Clara is the same character she played in Asylum of the Daleks remains to be seen - with guest stars Richard E Grant (Withnail & I) and Tom Ward (Silent Witness).

# like Have I Got.* News For You after Season 44 episode 09
00 21 21 12 * root /home/iain/bin/record 'Have I Got News for You s44e10 Christmas Special on BBC One for 31' # Actor Daniel Radcliffe, perhaps still best known to most as boy wizard Harry Potter (despite having laid down his wand 18 months ago) is tonight's host, taking charge of Muggle team captains Ian Hislop and Paul Merton, and panellists Andy Hamilton and Sara Cox.
45 23 25 12 * root /home/iain/bin/record 'Have I Got a Bit More News for You s44e10 Have I Got a Bit More News for You Christmas Special on BBC One for 46' # Actor Daniel Radcliffe, perhaps still best known to most as boy wizard Harry Potter (despite having laid down his wand 18 months ago) is tonight's host, taking charge of Muggle team captains Ian Hislop and Paul Merton, and panellists Andy Hamilton and Sara Cox.
30 19 27 12 * root /home/iain/bin/record 'Have I Got News for You s44e11 Have I Got News for You 2012 on BBC One for 31' # A compilation of highlights from the past year, remembering how Paul Merton, Ian Hislop and a variety of guest hosts and panellists took on the big news stories of 2012. Last in the series.

# Homeland after Season 02 episode 11
00 21 23 12 * root /home/iain/bin/record 'Homeland s2e12 The Choice on Channel 4 for 91' # Feature-length conclusion of the second series of the Emmy award-winning thriller. Carrie thinks about returning to the CIA, but wonders if a career in the intelligence service is really for her, and Nicholas Brody meets with Faber and considers his family's future. Meanwhile, Saul is ordered to undertake a secret mission and Quinn makes a momentous decision. Last in the series.

# The Snowman and The Snowdog
00 20 24 12 * root /home/iain/bin/record 'The Snowman and the Snowdog on Channel 4 for 31' # Animated sequel to Raymond Briggs' classic festive tale The Snowman, telling the story of another youngster's magical Christmas. A boy's snowman and snowdog come to life at the stroke of midnight and take him on an adventure to the North Pole, where he and his new companions meet an assortment of colourful characters, including Santa himself, before returning home - where a wonderful surprise awaits.

# The Royal Institution Christmas Lectures
55 02 27 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e1 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Air: The Elixir of Life on BBC Four for 61' # Peter Wothers explores the scientific elements using a periodic table made from audience members at the Royal Institution in London, to help uncover what the medieval alchemists knew about the air people breathe. In his investigation, he reveals these elements can be used to control fire, defy gravity and harness the power of a lightning storm.
00 20 27 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e2 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Water: The Fountain of Youth on BBC Four for 61' # Dr Peter Wothers investigates whether drinking water can restore his youth, and discovers how exploding balloons could help solve the energy crisis. The presenter is also joined by Paralympic gold medal-winning cyclist Mark Colbourne as they try to find out what happens when two of the most reactive elements on the periodic table, caesium and fluorine, meet.
00 20 28 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e3 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Earth: The Philosopher'\''s Stone on BBC Four for 61' # Dr Peter Wothers explores the elements within the earth and investigates whether it is possible to extract the world's most valuable minerals from them. He discovers how carbon dioxide can be turned into diamonds and attempts the ambitious feat of turning lead into gold. Joined by Professor Sir Harry Kroto, the pair find out what happens when you set fire to a diamond, and establish whether a member of the audience is worth their weight in gold. Last in the series.

Adding a comment line with the show name and previous episode makes it reasonably human-readable and tacking on a comment at the end of the cron line allows you to add in the episode synopsis. Works pretty well.

Smoothing the process

This is nice, but unless you have file name OCD, you won’t have a nice neat folder of TV shows. That’s why in the source of some of those scripts there’s reference to show_organise.php. This looks at any video file in a target folder, parses out the name, episode number and subtitle and then moves it into the TV shows folder in a sub-folder with the name of that show. It then writes a symlink to that file back to a “queue” folder so that there is one location to look for unwatched TV shows. Then is looks at the database file for XBMC and checks which of the shows in the queue folder have already been watched and it deletes the symlink for those shows. The queue folder now represents an unattended churn of just shows that haven’t been watched.

No one likes TV ads

Watching recorded TV is great for being able to fast-forward over the ads, but who wants to have to put their drink down and reach for the remote to do that? I found a reasonably reliable solution for skipping UK TV ads on the mythtv wiki. This ignores most ad detection tactics – looking for a network logo in the corner etc and takes advantage of a feature of UK commercial TV: All ad breaks are preceded and ended with tiny block of silence. The example script then takes the audio from a recording and uses mp3split (designed for splitting continuous audio into separate tracks) and uses the position of those splits to determine which are ads and which are the actual TV show. I tweaked this script and included it at the end of my record script so that it outputs edit decision list files which XBMC will automatically pick up and use in order to skip over ad breaks with no user intervention. Sweet, sweet sanity.

Remote schedule

This isn’t quite Sky+ though is it? I mean, with Sky+ you get an app which lets you schedule TV shows to record from your phone. You can however, get pretty close with a bit of Dropbox (or cloud storage service of your choice) trickery.

Step 1: Add your subscriptions file to Dropbox. I’m using the linux CLI client.

Step 2: Symlink from that file back to where the scheduler is looking, e.g. ln -s ~/Dropbox/subscriptions.txt /share/tv/subscriptions

Step 3: Monitor that file for changes. inotify is the swiss army knife of scripting when it comes for monitoring for file changes but it’s less straightforward if you want to set up what is effectively a service running constantly against a file. I did find a neat little script called when-changed, however which simplifies this massively. I just then have it set up so that whenever my subscriptions or listings file is changed, it re-runs the scheduler.

This is the command which I have set up:
/home/iain/bin/when-changed /share/tv/subscriptions /share/tv/listings.xml -c "/home/iain/bin/tv_priority_schedule | tee /etc/cron.d/tv-schedule /home/iain/Dropbox/tv-schedule.copy.txt" &

This checks the subscriptions and listings files for changes and runs the TV scheduler if it finds any. The output is a crontab file which it writes to cron.d but also creates a copy in my Dropbox. Now I can edit my listings in Dropbox on my phone and see the resulting TV schedule update a short while later, also on my phone.

Of course, if I wanted I could also have it automatically compress recordings and have them dropped into my Dropbox as well for remote watching, but I think that’s for a time when I have a fatter upstream…

Quick And Dirty DVR

October 26th, 2011 by Strawp


I recently dug out an old USB 1.1 Digital TV Tuner – a Hauppauge WinTV Nova-T USB, which I think I bought in about 2003 and eventually gave up on due to poor reliability under Windows, the crappy TV signal quality in Coventry and the success of excellent torrent sites like UKNova. Well, I’ve moved house now, and with an increased TV signal strength also came the bad news that I appear to be on a limited bandwidth ADSL line. I envisaged a single evening of plugging in the tuner, installing MythTV under Ubuntu and having a neat DVR to use.

Sadly, this was not to be the case.

Getting the tuner recognised under linux wasn’t too hard. The required firmware was already present in Ubuntu’s repositories but I couldn’t get the thing to scan. MythTV couldn’t open or ID the card and “scan” resulted in nothing. I even tested with the intended packaged drivers under Windows and got about as far.

Eventually, I found “w_scan” which does the kind of full-frequency scan your TV would do and was able to produce a channels.conf file in the format that tzap uses. Success! On the tzap page of the MythTV wiki it shows how you can use tzap to tune the device and “cat” to just dump the MPEG stream to file. Excellent – time for a quick and dirty script!

I then knocked up a “record” script, which takes easy-to-read commands like “record Eastenders on BBC ONE for 30″, tunes the card and dumps the MPEG stream to a sensible location. Combine that with some cron and I’ve got a hacky little DVR. XBMC can do the front end stuff.

I now have to get used to the idea of knowing I want to watch something before it airs, like we used to do in the 90s.

Here’s my TV shows crontab as an example:

# Record soaps off the TV
29  19  * * 2,4 /home/iain/bin/record Eastenders $(date +\%F_\%A) on BBC ONE for 32
59  19  * * 1,5 /home/iain/bin/record Eastenders $(date +\%F_\%A) on BBC ONE for 32
30  18  * * 1-5 /home/iain/bin/record Hollyoaks $(date +\%F_\%A) on Channel 4 for 27

# Watchable stuff
59  20  * * 5 /home/iain/bin/record Have I Got News For You $(date +\%F) on BBC ONE for 32
59  21  * * 5 /home/iain/bin/record QI $(date +\%F) on BBC TWO for 32
0   18  * * 1-5 /home/iain/bin/record The Simpsons $(date +\%F) on Channel 4 for 30

Or for one-offs (as root, unless you change permissions on the device):

echo "record some tv on bbc one for 25" | at 16:00

(or as commandlinefu.com would probably prefer it:)

at 16:00 <<< 'record some tv on bbc one for 25'

Let me know in the comments if you end up using it.

DIY Hands Free Kit

May 10th, 2011 by Strawp

A few years back I came up with the CarPC as a cheapass hacky way of getting MP3 playback integrated into my unspectacular car. Since then, the world of mobile has moved on a whole lot and the laptop stashed in my boot actually pales in comparison with the features (and power) of my Desire Z Android phone. Plus, I’m already syncing music with my phone, like I used to with my car PC and with Google Navigation I have a powerful satnav solution already at hand. All I really needed at this juncture was some kind of hands free setup.

Bluetooth control?

Grom Audio, who made the CD autochanger interface that I used for my car PC also make an autochanger interface which extends with a bluetooth module. This seemed like an unnecessary expense however – I already had the Car2PC autochanger interface which still acts as a high quality stereo input even without the USB control via the car head unit.

The official HTC headset

The headset that came with my phone has an integrated mic with three buttons: answer call/play/pause, next track and previous track. Wouldn’t it be cool if someone had worked out what the circuit inside that thing was? I googled around a while but in the end, one of my friends pointed me to George Smart’s Wiki, where he’d already sacrificed his headset and worked out a circuit diagram. This was massively useful, and given that the circuit consisted of a few cheap components, I thought I’d see

if I could give it a go and make a working copy.

Circuit prototype

If, like me you haven’t done any electronics since you were at school, it’s a good idea before you start soldering to get a prototype board. This is the Lego of electronics – you can stick components into the board without solder and switch things around if needed. I got the smallest prototype board I could find from Maplin for a couple of quid and had a circuit which gave the right resistances fairly quickly. The idea was to produce something I could plug my phone into on one end and would have stereo output and a mic connection so that any generic mic with a 3.5mm plug could be used with it. In addition to the components list on George’s wiki I needed an extra 3.5mm microphone socket, a 3.5mm stereo socket, a 3.5mm 4-pole plug (for connecting to the phone).

Now, it turns out the connecting to the phone part was problematic. The audio connection on HTC phones is pretty much the same as an iPhone. It’s compatible with a standard stereo heaphone connection (3.5mm 3-pole plug) but it has a third “control” ring on the plug which is used to carry the signal for the microphone. Shorting this signal (with the play button) sends the signal to the device to play music and the next/previous buttons simply send approximate voltage signals down the wire by varying the resistance slightly. In order for the controls to work I needed a 4-pole plug. You can get 3.5mm 4-pole plugs from Maplins, but holy crap are they fiddly to solder. I spent a couple of really frustrating hours trying to get four lots of wire to stick to the really unhelpful contact surface on the plug I bought. These were not designed for people whose GCSE in Technology a distant memory.

In the end I gave up and managed to find a much more solder-friendly 4-pole socket on eBay and got a pre-made male-to-male 4-pole plug cable (which you can even pick up in HMV these days).

Two resistances and a microphone

Here’s the part where I’m glad I used a prototype board first. I selected a reasonable looking tie-clip mic online that would handle calls and voice commands for the phone and plugged it into the circuit. Suddenly, the “next track” button didn’t work. What had transpired was that by adding what was effectively a 1.4 kilo-ohm resistor to the circuit in parallel, I’d lowered the effective resistance when I pressed the next track button. This meant the phone interpreted it as “previous track”. To work around this, I needed to raise the resistance on the “next track” button. If you are trying this and get the problem, my method was to measure resistance across the mic with a multimeter, then using the resistors in parallel equation plug in the target resistance on the control wire (560 ohms) and the value of the other parallel resistor on the circuit. With a little help from ElectroDroid, this gave me a new resistor value to plug in (1 kilo-ohm, I think I went for) which then made all the buttons work with the mic attached.

Constructing the device

I guessed an approximate size for the circuitry, buttons and sockets and got a small project box from Maplin. For my first attempt at this, I figured I’d make life easy for myself and just put all the connections on top so tha

t I could work on cutting the holes to the right shape and getting everything to fit in, then I could just pop the circuit board into the box and screw the lid down.

The photo is what I ended up with. From the left, the interface is:

  • Stereo audio out (to the car audio system)
  • 4-pole 3.5mm stereo audio and control I/O (the phone)
  • 3.5mm 2-pole mic socket (goes to tie-clip mic, which is going above the door frame in my car)
  • Previous track button
  • Play/answer button
  • Next track button

Seamless automobile integration AKA “The part with the duct tape”

So that’s all the bits working. Now to find a space in my car that I can fit this where it’s in easy reach, won’t get in the way and can be secured easily. In the end I went for the space just in front of my gear stick, mounting it next to the 12v power socket, over the top of the ash tray. The stereo cable for the car audio actually goes through the back of the car and plugs into the Grom Audio CD autochanger emulator. The mic cable is taped around the dash up the side of the windscreen and sits near the top of the door, in clear audible range. The phone is held in a generic phone mount on the top of the dashboard and plugged in to the 12v power (converted for the phone of course).

I might neaten it up. I probably won’t.

Extending the controls

Having the standard audio controls, plus the ability to take calls is all well and good, but what about being able to do other useful stuff with those 3 buttons? I’ve previously blogged an example of using Tasker with the Headset Button Controller app, so I won’t labour the point, but suffice to say with these two little gems you can pretty much tie or automate any Android task you need to with only 3 buttons at your disposal. I keep my Tasker profile checked into SVN if you want to borrow bits of code, and I’m changing it all the time but here’s some of the things I’ve used this setup for:

  • Press a button to activate voice commands (“navigate to cambridge”, “call home”, “listen to beastie boys” etc)
  • “Love” the currently playing track
  • Automatically start playing music when headset and power is attached (poor man’s dock mode)
  • Automatically read out incoming texts using text-to-speech
  • Announce callers using text-to-speech
  • Re-read the last text message
  • Call a specific person using a button press
  • Launch navigation to a specific place with a button press (i.e. the “oh shit, I’m lost” button)
  • Lock onto heat signatures and engage stinger missiles

So there we go. That’s the kind of car setup that you’re only just starting to see in new cars at the moment (at time of writing, ahem) and for complete pocket money.

“Love” tracks on last.fm with one click of a headset button

April 30th, 2011 by Strawp

Here’s a handy little trick for keen last.fm users on Android that lets you mark the current track as “Loved” with a headset button combination. What you will need is:

  1. The official Last.fm client for Android
  2. Tasker
  3. Headset Button Controller
  4. A headset remote button or the headphones that probably came with your phone if it’s got buttons on it

Set up last.fm

Nothing much to do other than log in and make sure your media player of choice scrobbles tracks correctly. I’m using Winamp which works fine, but make sure you’ve turned on scrobbling in the settings. The last.fm widget has a “love” icon which due to the awesomeness of Android’s intents architecture we will be calling on. Add the widget somewhere on your home screen and make sure that bit works.

Create a “Love Track” task in Tasker

In Tasker, click “Tasks” -> “New Task” (call it “Love Track”) and in the task edit window add “Misc” -> “Action Intent” and in the “Action” setting, enter “fm.last.android.widget.LOVE”. Leave the other fields blank and make sure “Target” is set to “Broadcast Receiver”. You could also add an action which says “You love this track” by using “Misc” -> “Say”.

Start a track playing and come back to the edit screen and then hit “Test”. The last.fm app should create a little toast notification (one line of text overlayed on the screen) saying the track has been marked as played. After a short while it should appear in your last.fm profile.

Optional: Create a home screen shortcut

Long-click on a space on your Android home screen and choose “shortcuts” -> “Task” -> “Love Track”. You can now click on that to mark the currently playing track.

Headset button controller

Headset button controller is a stupendously useful app that allows you to assign actions to headset buttons. If your headset only has one button this is great because you can do more than answer/hang up and play/pause by assigning double, triple, quadruple and long-click actions to a button.

Once you’ve installed this set up your usual button actions, simply set one to “Tasker Task” -> “Love Track” and you’re done!

Remote Wireless Music Syncing Android and Linux

February 13th, 2011 by Strawp

Recently, an Android version of Winamp was released and included in its feature list was very useful wireless syncing with the Windows version of Winamp. Unfortunately, my file server at home doesn’t run Windows and Winamp under Wine is an unstable mess, so attempting to use that feature was out. Like a good Linux user, I didn’t take this lying down – we don’t sit around like Windows users waiting for someone to build the solution for us, we cobble something together using tools already available! What I decided to do was to emulate the mechanism that I came up with when I built my Car PC. Basically, the core of wireless syncing in the Car PC was rsync, combined with a little bit of logic to get a list of music track locations to pull over. By offloading some of that logic onto the server, this seemed possible.

First, I installed an rsync app for Android and tested it out. Rsync backup is just a front end for rsync, which is perfect because it lets you use all the normal rsync options, allows public key authentication (so you never need to enter a password) and displays rsync’s output to a log window. I wanted to pull files from my server so in my rsync profile I selected “rsync in reverse” and added the command line options “-vHrtDL --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --progress --partial” most of which is the default for a new profile, but I added:

  • -L, to follow symlinks
  • --progress, to display file transfer progress
  • --partial, to allow file transfer resuming

Android won’t know any of the new music files have been added yet, so they won’t show up in any music players (which all look up available tracks in the built-in Android media database), so you need to install something like SDrescan to run after the files have copied over which will magically make them visible in the music app of your choice.

That’s the phone side sorted out, I just need something sensible to point it at – I don’t want rsync recursing over my entire music collection wirelessly, it would take ages.

On the server, I’m running the (somewhat ageing) Firefly Media Server, which indexes and serves music for my Roku Soundbridge. I’d previously written a data abstraction class in PHP for the sqlite database it runs on, so I used that to produce a script (to run periodically under cron) which picks a list of albums from a playlist (created by another script) and creates a bunch of symlinks to them in the folder that I set as the target in my rsync profile, the idea being I have a regularly changing folder which one rsync command can look at to pick up new music.

I now have wireless syncing that works anywhere with an internet connection, as well as on the local network (unlike Winamp) and the idea of keeping a static rsync profile and using a source folder full of symlinks provides a cunning way of pushing content onto your phone. It would be quite easy to create something like a web interface to pick files on the server and have symlinks of them created on a single rsync source folder. On top of this, Rsync backup provides hooks for Tasker to link into, so you could set this up to run automatically at some opportune moment.

How FireShepherd could live up to its name

November 3rd, 2010 by Strawp

First there was FireSheep. It allows anyone to hijack HTTP session cookies for a number of sites for anyone using them on the same open access point as you. Now, a predictable counter point for that is that someone would come up with “FireShepherd” to protect this poor flock. However, FireShepherd is no where near as fun as FireSheep – all it does is try and crash FireSheep with fake data and hope for the best, meanwhile your session info is still being transmitted in the clear.

Ideas for FireShepherd to be more useful/fun:

  1. Have it force SSL connections on all the same sites that FireSheep snoops on, making session hijacking impossible. Plugins like Force-TLS do this.
  2. Have it create bogus logins to sites where the user’s profile pic has been set as goatse, tubgirl etc. When the FireSheep user grabs that user’s session data they will have those lovely pics appear in their stolen sessions list.
  3. (getting crazy here) have it perform a man-in-the-middle attack on the wireless network, replacing the network’s router as the default gateway or DNS server. You can then point people to fake versions of captured websites and feed the FireSheep user whatever you want. Oops, there’s goatse again! Oh, what’s that you just went to? A malware site? Careless FireSheep user!

Anyway, there’re some ideas. As Steve Gibson pointed out in the last Security Now, simply switching a network to WPA is enough to protect all the users from this attack. If you’re running a Cafe and want to provide free wifi you can make the network password as public as you want – make a poster and stick it above the till. It is unencrypted wifi, not wifi itself that allows user sessions to be hijacked like this.

Proof of concept: Locating a remote machine using the Google API

October 27th, 2010 by Strawp

example output of the locate script

A couple of months ago Samy Kamkar presented a cool hack at the Black Hat Conference which demonstrated that by using a cunningly constructed URL against specific internet routes, you could inject some javascript on their configuration pages and trick them into sending their own MAC address (the router’s own unique code) to a script which would use it to look up their address on Google’s API, thus telling the attacker a pretty good estimate of your physical location.

This was only possible because as well as photographing everything, the Google Streetview cars have been recording the locations of every wireless access point they encountered. By recording the signal strengths of certain access points by the location they were observed in it’s possible to do a simple triangulation calculation and get a pretty good estimate of where the access point is.

I found this pretty fascinating, so I created a little PHP script to use this trick, perhaps it could be used to help locate a stolen laptop, for example. The script works like this:

  1. Attempt to scan using the wireless network adapter for any nearby access points. Save their key details (BSSID, signal strength and most importantly, the MAC)
  2. Compile all this into POST data and send it to Google using libcurl. Even without nearby MACs Google’s location API will do a better job at locating the machine than the usual GeoIP services.
  3. Get a bunch of data back from Google, including longitude and latitude estimates and a street address. For ease this also gives you a Google Maps link

The accuracy depends on how many neighbours your wireless card picks up and how much data Google has harvested from the wireless networks on your road, but for most people it will be accurate almost to the right house number.

The script is on my SVN server for download

How to download your Facebook photos

October 27th, 2010 by Strawp

Facebook recently released a feature that allows you to download a static version of your Facebook profile which includes all your videos, photos, your wall and a few bits of periphery information. If you’re planning on quitting Facebook, this is great because it means you don’t lose anything and it’s also a nice offline backup of your Facebook info.

To do this, go to Account -> Account Settings and pick “Download Your Information”.

What this doesn’t get you, however is all the photos you’re tagged in. For this you need to go through the Facebook API. Here’s how to grab all those photos in the best possible quality in a bit of a hacky way. No idea if it violates Facebook’s terms of service – who the hell knows what they are any more?

First of all you’ll need a system which has wget, grep, sed and awk. You can get these for Windows and this should all work, but I’m going to assume you’re logged into something with a bash prompt.

  1. Find your Facebook ID. If like most people you have an alias for your homepage, click on your profile picture and you’ll see “id=xxxxxxxx”. That’s your Facebook ID
  2. Create a URL for calling photos.get in the API. Go to the API documentation page for photos.get and in the Test Console at the bottom, enter your Facebook ID for “subj_id”.
  3. Run the call. Click “Call Method” – you will then see a bunch of code on the right and you will have a URL at the top starting with https://api.facebook.com/method/photos.get?subj_id=. Click on it to open that page in a new window.
  4. Download the data. Either copy the file from the browser window (ctrl+a, ctrl+c, paste into a file) or run wget -O photos.json "<your URL>" to save your data in a file called photos.json
  5. Grab the images. You can now run cat photos.json | sed 's/,/\n/g' | grep src_big | grep http | sed 's/\\//g' | awk -F\" '{print $4}' | wget -i -. This will pluck out the URL of each photo you’re tagged in and download them to the current folder.

You’ll then see wget going crazy downloading all those photos and when it’s done you’ll have a copy of every photo you’ve been tagged in. After that you’re free to stick them into an album, edit them into your existing offline Facebook profile – whatever you fancy.

Recipe for a decent bash shell in Android

August 11th, 2010 by Strawp
  1. Root your phone if you haven’t already and install the latest stable CyanogenMod, which includes bash. If this step sounds scary, stop reading now :)
  2. Install ConnectBot:
    Connectbot Market QR code
  3. ConnectBot is a fantastic SSH client, but it also has a local terminal emulator. Open it up and select “local” from the drop down list and give it any nickname.
  4. Open it up to see the local shell works. It will, it’s not bash though and it won’t have tab completion. Hit the menu button and disconnect.
  5. Long-press the connection and select Edit host.
  6. Find the “Post-login automation” option.
  7. enter:
    su
    bash
    export PS1="\w\$ "
    cd /

    This step automatically switches you into root mode (you will be prompted by the superuser manager the first time you do this), starts bash, sets your prompt to the working directory and then changes directory to root.

  8. Hit “OK” and then the back button to return to the connections screen. Select the connection to test if it’s all working.
  9. For quick access, you can add a shortcut to the home screen. Long press on the home screen, select “Shortcuts”, “ConnectBot” then the name of your connection.

Tada! A one-click root terminal in bash with tab-completion (courtesy of ConnectBot’s keyboard shortcuts). Much better than the Terminal app every other blog tells you to install ;)

OAuth PHP Twitter Class

June 22nd, 2010 by Strawp

After posting my temporary twitter unfollow script the other week I was reminded that Twitter is supposed to be dumping basic auth (sending user name and password in each request to Twitter) this month in favour of the more secure, widely adopted Open Authentication. It turned out Twitter has postponed what they have dubbed the “oauthcalypse” due to the world cup but if you haven’t already, now is the time to get rid of your neat little one-line curl scripts using basic auth (a shame I know but it’s for the best) and find an OAuth alternative in the language of your choice.

I have created a simple TwitterClient class (available under SVN) based on my previous set of basic auth and curl-using functions from a couple of years back. I have left it with my app key in their on purpose, so tweets from it will appear to come from “Strawp’s PHP lib”. Register your own app if you want and use those instead, of course. On the first use of a method you will be prompted to authenticate the class against your twitter login. The secret keys will then be stored in your home folder.

Example usage:

include( "twitter.class.php" );
$tw = new TwitterClient();
$tw->tweet( "I am using oAuth now!" );

The unfollow script has also been updated to use this class.