What I’ve Been Up To: H4xing and Pwning

December 9th, 2015 by Strawp

I occasionally touched on security topics in this blog. The last two posts were about the Sophos Blackhat 2013 puzzle and Greedy Git. I guess one thing I didn’t post about, which had massive knock-on effects was my involvement in the excellent Cyber Security Challenge UK, which I entered on a whim in 2011, played for 2 years (getting to the Masterclass Final in both) and had loads of incredibly fun and useful experiences with other enthusiasts and security professionals. One of the prizes from the competition was the Certified Application Security Tester (CAST) course from 7Safe, which was great and helped me loads with making my web application development more security focussed. Ultimately though, I found that breaking other people’s web sites is way more fun than the painstaking process of building your own and I decided to pack in being a web developer and become a pentester, getting my CREST Registered Tester qualification in my own time (so that someone would take me seriously) and then getting a job at Nettitude. About 9 months later I upgraded my CREST qualification to the web-centred “CREST Certified Tester (Applications)” (CCT).

Most of what I do day to day I can’t talk about in detail, but here’s some stuff I’ve blogged for Nettitude to give you a general idea:

Additionally, I tend to post relevant security stuff to my Twitter, @strawp and put little code projects on my GitHub (and Nettitude’s GitHub).

Blackhat 2013 Sophos Puzzle

August 7th, 2013 by Strawp

Recently, Sophos like to mark attendance at a trade show by running a geeky puzzle competition and they’ve just done one for blackhat, which I actually managed to finish.

A hackable crossword puzzle

The puzzle started with a fairly straightforward crossword. Now, I’m terrible at crosswords, but fortunately this crossword had an answer checking function in its Javascript source code which contained hashes of the answers, using its own simple hashing function:

AnswerHash = new Array(81594, 21864, 31173, 82603, 59177, 71467, 9693, 97258, 
  92537, 49915, 30431, 16251, 59242, 66492, 67266, 96088, 2956, 97195, 73488, 
  73937, 38947, 81055);
// Returns a one-way hash for a word.
function HashWord(Word)
    var x = (Word.charCodeAt(0) * 719) % 1138;
    var Hash = 837;
    var i;
    for (i = 1; i <= Word.length; i++)
        Hash = (Hash * i + 5 + (Word.charCodeAt(i - 1) - 64) * x) % 98503;
    return Hash;

Porting this into PHP (my default scripting language) and using /usr/share/dict/words, it was pretty simple to solve a good number of the questions automatically. The hashing algorithm was prone to a lot of collisions, but narrowing down the possible words by length helped. I checked this worked properly against an easy clue: “How information starts its life”, which of course is “data”.

This technique gave me a nice spread of completed words across the board:

    • Decrement RCX and branch if not zero.: loop
    • The moves a pentester makes once he’s in.: lateral
    • Autonomous software (but not quite a virus).: agent
    • It’s not a lens, but it’s focused on you anyway.: prism
    • Vulnerabilities that really work.: exploits
    • How information starts its life.: data
    • On the Mac, it’s Option E.: acute
    • Where you set up the base pointer.: prolog
    • The guy who’ll win in the Apple-Samsung case.: attorney
    • What amateur cryptograms are always claimed to be.: airtight
    • Apple couldn’t bring themselves to call it Wi-Fi.: airport
    • Whitfield Diffie helped you share it.: secret
    • What the BlackHat trade show staff are really after.: leads

Then spotting a few more easy ones: “You’ll read it if you want to win the prize. [5,8]” is of courseNaked Security“, “What you do to your code when you’re in a hurry.” I got worryingly quickly: “Hack at it”. A few more required some googling: “He decrypted Hittite” is Hrozny. Curiously the last clue to get was “Why you are doing this puzzle”. “it’s fun” and “I’m bored” didn’t fit in, but I had enough letters that I could come up with possible combinations of words based on the words list and grep. You can do this with any crossword e.g. the first word “_H_E_” can be found with:

grep -i "^.h.e.$" /usr/share/dict/words

and the last word with

grep -i "^.r.n.e.$" /usr/share/dict/words

Despite the middle letter being “d” and “three” coming up in the first list, I still didn’t spot the answer (see, terrible at crosswords) so I combined the 26 possible first words with the 45 possible last words, creating a list of 1170 possible words, which I could then just run the hashing function against to reveal the right answer. This made me feel simultaneously cunning and stupid.


The completed crossword


Moving on…

Counting in binary

As per the instructions on the competition page, this then gives you a string of 6 letters which form the password for the zip file containing the next stage, however you don’t know the case of each letter, which effectively means there are 2^6 possible passwords for the zip file. By considering an uppercase character as 1 and a lowercase character as 0, this is basically the same as counting from 0 to 64 in binary. e.g. if the possible letters were ABCDEF, you would first try abcdef, then abcdeF, abcdEf, abcdEF etc.

I wrote a simple script to create the word list and then ran “unzip -P [word] snodwen-message.zip” against each word. This gets you to the next stage.

Oh God, not FORTRAN

I hadn’t seen a line of FORTRAN for 12 blissful years until this point. The not-so-subtle reference to current affairs was a message as follows:

Dear Reader,

This is Teddy Snodwen speaking.

You don’t know me, and I don’t know you, but we may be able to help each other.

I have some private data I’ve encrypted, but I’m having some travel problems right now, with the result that I’m concerned about getting stuck in no-man’s land at some airport, unable to leave, or proceed, or get at my data.

So I have prepared a series of files from which anyone who’d like to help can extract a secret code that can be read out over the phone, or even just held up to the glass in the transit area for me to copy down.

When you’re ready, you’ll need a PDF file from here:


And you’ll need the password, a nine-digit number you can calculate with this simple algorithm, which I’ve written in my favourite programming language, MR-ISP.

It’s a rare dialect of FORTRAN:

DO 51 I=1,1000000000
DO 52 I=1,9
S=S+(INT(P*(10**Q)) MOD 10)

I know you won’t so much as think of cracking the code until I give the signal, since gentlefolk don’t read other gentlefolks’ email.

And with that, I remain,

Yours sincerely,

Teddy Snodwen (Mr)

MR-ISP, eh? Subtle.

With some minor tweeks, that just about runs in FORTRAN, however unfortunately what you will very quickly notice is that the code is dealing with stupendously large numbers and a stupendously large precision number. The code is basically split into two parts:

Part 1: Iterate 1 billion times over the equation for phi to have that value to a very high degree of accuracy.
Part 2: Select digits from the decimal places of phi, in orders of magnitude increasing by 10. i.e. 1st decimal place, 10th, 100th, 1000th etc.

This doesn’t work in pretty much any programming language unless you use a library with which you can specify arbitrary precision of your numbers. So there are two ways of solving this:

  1. Find an arbitrary precision library for FORTRAN, get it to work with the above code, run the code
  2. Some other way which means I don’t have to touch any more FORTRAN or start looking for arbitrary precision libraries in another language

Obviously I chose #2. I can’t have been the first person on the internet to calculate phi to a ludicrous degree of accuracy. Indeed not. Someone had also written a program called y-cruncher which calculates famous constants to arbitrary precision and outputs them to a text file. Much nicer. I could then just pick out the digits by hand and come up with the 9 digit password for the PDF. Next.

Onion Skins Of Encryption

The Unlocked PDF has the URL and password for the next stage – another zip file. This contains a single text file – “e.9” – which contains Lua code. It’s clear from the first few characters that the code is an array of data encrypted using an XOR at some point:

k=11179023 o=bit32.bxor t=string.char f=math.floor c={
} function xit(n) local x=o(11994318,n) for i=1,24 do x=x*2 if x>=2^24 then x=o(x,25578747) end end return x end
function tit(n) return t(n%256)..t(f(n/256)%256)..t(f(n/65536)%256) end
p='' for i=1,#c do p=p..tit(o(k,c[i])) k=xit(k) end load(p)()

The code takes a very large array of integers, then iterates over them, XORing each integer against a key, then splitting the result into 3 bytes, adding that to a string and then creating a new key for the next iteration by shifting the old key one binary place to the left and then XORing that against a fixed number. Once all the iterations have finished the resulting string is loaded as Lua code and ran.

The first thing you would do once getting to this stage is to just run the code as is. You quickly find that:

  1. It takes ages
  2. It produces garbage

Clearly you need to change a few things. First of all, you don’t need to iterate over the entire array to find if it’s worked or not. Iterate over just 10 values by switching “#c” for “10” and switching “load(p)()” for “print(p)”. The next thing you have to do is make a few assumptions. What’s broken? The algorithm looks OK, so maybe the initial key isn’t right. To find a key for an XOR encrypted text is fairly easy if you know what the unencrypted text is for a part of the cypher text larger or equal to they key. The key is only 3 bytes long (only 3 characters), so this shouldn’t be too hard. I hadn’t used Lua before but to me, it looked like the last line “load(p)()” was treating “p” like a function and running it. Maybe the first line contains a function definition? I tried XORing the first 3 bytes of the cypher text against “fun” as in “function” using this code added to the end of e.9, replacing the normal decrypt loop:

function untit(test)
  return string.byte(test,1)+string.byte( test, 2 )*256+string.byte(test,3)*65536
k = o(c[1],v)
p = ''
for j=1,#c 
	print( j.."/"..#c )
	if j>10 and not string.match( p, "function" ) then

This provides the reverse function of “tit”, “untit” and attempts to start the decryption by setting the initial key as the result of XORing the first 3 bytes with “fun”. If after 10 iterations the resulting string doesn’t contain “function” then the loop exits. If it does, then the decryption has worked and it continues to decrypt the rest of the array.

This didn’t work. The decrypted text clearly didn’t start with “fun”. So what then? The clue is the filename, “e.9”. The “e” is probably for “encryption” and the “9”? How about the 9th iteration? Suppose the file decrypts another selection of encrypted text? If it used the same algorithm that would mean the file started with “k=” and then a number between 0 and 9. Only 10 possible combinations of clear text for the first three bytes! I then took the above code and put it in a loop:

m = "k="
for i=0,9
  test = m..i
  k = o(c[1],v)
  p = ''
  print( i )
  for j=1,#c 
    print( j.."/"..#c )
    if j>10 and not string.match( p, "bit32" ) then
  if string.match( p, "bit32" ) then 
    print( test..": "..untit( k  ))
    file = io.open( "e.8", "w" )

This attempts to decrypt using the known plaintext of “k=0”, “k=1” up to “k=9”. It then lets the algorithm carry on for 10 iterations and then tests to see if “bit32” has come out in the resulting string, as we know that this is also at the start of the code. If it has, it carries on decrypting the code and then writes it to file “e.8”.

As suspected, e.8 is basically the same, but slightly smaller. I could then just replace the array in my code with e.8’s, run it again and find e.7. Repeating this, you eventually get down to “e.1” and it stops working, suggesting that “e.0” doesn’t have the same code in it. We need a new known plaintext. Assuming e.0 was still Lua code, I tried a few things, eventually finding that it starts with “print”.

Extreme checksum

This is the final message:

In stage 1, you solved a crossword, extracted 24 characters 
from the completed grid, and used six of those characters to 
form a password.

Now take the remaining 18 characters and write them down in 
reverse alphabetic order (Z..A).

Then write a dollar sign.

In stage two, you calculated more than 400,000,000 decimal 
places of a certain transcendental number, and used nine of 
those digits to form a password.

Now write down the nine digits from decimal places 
100,000,001 to 100,000,009 inclusive.

Then write a dollar sign.

Now write four alphabetic characters (A-Z) of your choosing.

You should have a string of 33 characters. Ensure all letters 
are upper case.

Calculate the 512-bit SHA-3 hash of this string, print it as 
hexadecimal characters and use the first 20 as your answer.

Submit your answer as detailed here:


Fun! Basically, “You got this far, now prove again that you didn’t cheat”. If you were diligent in the early stages, you will have saved all your answers as you went along and this won’t take long. I took a screenshot of my crossword, thankfully but failed to save my digits of phi, so I had to do that bit again.

What’s with the last 4 characters? The puzzle author will check your hash against a rainbow table1 of 26^4 possible combinations to check you have the right answer, and presumably to restrict sharing of the hash.

Thanks to Paul Ducklin for the puzzle. It had a really nice difficulty curve that drew me in with an easy crossword and before too long I was writing in two programming languages I usually don’t touch.

[1] From the author:

“Less of a rainbow table and more of a list…only 26^4 options (about 0.5M).”

, however I consider anything I wouldn’t want to write by hand a rainbow table. Shopping list, you can do manually. Shopping rainbow table would be very expensive.

Greedy Git

June 23rd, 2013 by Strawp

TL;DR: I wrote a script for enumerating and downloading source code off sites when they accidentally share their .git folder.

I’ve notice a couple of things of late when looking at security testing software:

  1. Python now seems to be the scripting language of choice
  2. It seems to be mostly in not just git, but on github

And the upswing in popularity of git is not just in small projects, but in deploying to web sites too, so much so that it’s now becoming increasingly more likely to find a site that is inadvertently serving the .git folder to the outside world.

With a little work it should be possible to reconstruct a repository remotely (object packs being the only hard part).

Of course, this isn’t a new problem – SVN has the same issue – but the fact that it’s slightly harder to parse that git metadata means it’s a nice opportunity to finally take the plunge and write some Python and learn more about git.

What’s in the .git folder?

  • an index file which, like SVN is effectively a database of all files in the project against hashes of those files. Unlike SVN, it’s in memory map format, which is much more fiddly to write code for
  • The entire site source code, reference by an SHA1 hash, compressed using zlib deflate
  • Logs of git actions such as commit in logs/HEAD
  • A small config file which is a good starting point to test if a .git directory is present or whether the site is configured to return 200 OK for any URL requested as it returns a very predictable format

Analysing the .git folder

As a starting point, to avoid having to parse the index file myself, I forked gin – a neat little index file parser written in Python. This already produces a readable and JSON encoded version of the file which I can then use to iterate over the files. The script looks at:

  • File extensions. Count which file extensions are the most popular. This tells us what our site is written in, if it wasn’t already obvious
  • “Interesting” files. Archive format files, backups, SQL, “hidden” files (beginning with “.”) such as .htaccess and .htpassword, files which might have DB configurations in them etc
  • The logs/HEAD file for emails and credentials stored in URLs

This then dumps this information out into a simple flat text interesting.lst file, a report.md file, containing the results of the above scan, a copy of index in its native format, readable text, json, and flat text and copies of config and logs/HEAD

Being a greedy git

At this point, you already have quite a lot of powerful info however if the script has managed the above, it will probably also be able to download the source code for the site. Since we’ve already determined a lot of interesting files in interesting.lst, we can use that (edit it and add to it) to download all those files to our computer. In git, the compressed source for a file (in “loose” format) is stored in .git/objects/ and is referenced by the SHA1 hash of itself. We have that hash, so we can try and download files.

Passing the “-I” command line argument to greedy-git will make it attempt to download everything in interesting.lst to ./files/ in the current working directory. If you really want to go overboard, you can pass “-a“, which will get as much of the site source code that it knows about, and passing “-g [remote/file/path]” will download just that file, or matching file pattern.

You now have a target site’s juicy source code. This could contain database or other credentials, clues to vulnerabilities or “security by obscurity” style back doors that the developer thought no one would find. All this is now just a few grep commands away.

Do use responsibly, and let me know if there is a way of guessing the pack file name – that would be the keys to the kingdom…

Quick and Dirty DVR Redux

December 19th, 2012 by Strawp

Dropbox is handy
It’s over a year since my blog post entitled “Quick and Dirty DVR” and I’ve been using and tweaking the code ever since. What I have now is a pretty autonomous and functional video recorder based around that core record script.

All code mentioned in this post can be found either in my scripts or lib svn repository. I apologise for the hard-coded paths…

What I’ve added

  1. Scheduling & collision detection
  2. File management
  3. Ad detection
  4. Remote scheduling

Automatic scheduling

The first thing was to get the script to find and record TV shows automatically. To do this, the script needs to know:

  1. A source of TV listings
  2. A list of shows that I watch
  3. The last episode number of a given show that has already been recorded
  4. When all subsequent shows are on, in the future

The first step is easy – just install the XMLTV package and set up a cron tab: 0 2 * * * /usr/bin/tv_grab_uk_rt --quiet > /share/tv/listings.xml. This will check TV listings nightly.

The list of TV shows is simply a flat text file – one show title per line. I took the decision to allow regular expressions in show titles when the show name starts with “like”. For example, QI and QI XL are largely interchangeable so the entry for QI is “like QI( XL)?”, meaning the “XL” part can be there or not but it will be considered to be the same show.

Then I just have to write something to work out which TV shows I already have. It’s very helpful to have your TV neatly organised for this. All my shows are in the path /share/tv/[name of tv show]/[name.of.tv.show].s[series]e[episode].[subtitle].[extension], so inside tv.func.php is the function getLastEpisodeOf($name,$dir) which returns text describing which episodes of that show should be recorded, e.g. “Peep Show after season 8 episode 2”.

Next the listings file needs to be parsed. For a while I had some very dumb parsing which went through each line of the subscriptions file and picked the first available episode of whichever show it was searching for. This was flawed as the scheduler wasn’t aware of other shows being recorded and therefore clashes in scheduling were inevitable. The most I could do was to detect the collisions between shows after the schedule was written and manually correct them.

After a bit of thought I replaced the scheduler with a new script which takes the order in the list of TV shows file as a priority for each show and checks that any new show attempting to be recorded does not clash with a show already scheduled. If this happens, alternative times are checked and used. The presence of “+1” channels makes this a lot more robust. If a show doesn’t have a suitable recording slot, it’s dropped from the schedule and a warning is produced in the crontab output as a comment.

This is an extract of my subscriptions file:

Fresh Meat
The Killing
Peep Show
Bang Goes the Theory
Doctor Who
Top Gear
The Simpsons
like QI( XL)?
like Have I Got.* News For You
The Snowman and The Snowdog
The Royal Institution Christmas Lectures

And this is an extract of how the recording schedule looks for the Christmas 2012 period:

# Clash: The Royal Institution Christmas Lectures s1e1 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Air: The Elixir of Life on BBC Four for 61 at 2012-12-26 20:00-21:01 with EastEnders s1e4556 2012-12-26 Wednesday on BBC One for 31 at 2012-12-26 20:30-21:01
# Clash: Tangled on ITV1+1 (Central) for 111 at 2012-12-25 16:10-18:01 with Doctor Who s7e6 The Snowmen on BBC One for 61 at 2012-12-25 17:15-18:16
# No suitable schedule found for Tangled

# Peep Show after Season 08 episode 04
30 22 23 12 * root /home/iain/bin/record 'Peep Show s8e5 Chairman Mark on Channel 4 for 31' # Jeremy is desperate to avoid living with Super Hans and rekindles his relationship with Mark's sister Sarah, moving in with her and her five-year-old son Joshy. In an attempt to deal with a damp patch on his bedroom wall, Mark launches a campaign to be elected chairman of Apollo House's freehold committee. Starring Robert Webb and David Mitchell. The final episode is tomorrow at 10pm.
00 22 24 12 * root /home/iain/bin/record 'Peep Show s8e6 Quantocking II on Channel 4 for 31' # Dobby is offered a job in New York by ex-boyfriend Simon, but is uncertain whether to accept the offer. She suggests to Mark they should go for a weekend break in the country, while Jeremy decides it is time to get a few things off his chest. Comedy, starring David Mitchell and Robert Webb. Last in the series.

# Doctor Who after Season 07 episode 05
15 17 25 12 * root /home/iain/bin/record 'Doctor Who s7e6 The Snowmen on BBC One for 61' # Matt Smith returns as the Time Lord, who is in mourning after losing Amy and Rory to the Weeping Angels and determined to avoid getting mixed up in the universe's problems. But a call for help whisks him back to Christmas Eve 1892, where a trio of old friends and a plucky young governess called Clara need him to take on a chilly menace that comes with the snowfall. Will he be persuaded to abandon his new life as a recluse and defend his beloved Earth once more? Jenna-Louise Coleman joins the Doctor as his new companion - although whether Clara is the same character she played in Asylum of the Daleks remains to be seen - with guest stars Richard E Grant (Withnail & I) and Tom Ward (Silent Witness).

# like Have I Got.* News For You after Season 44 episode 09
00 21 21 12 * root /home/iain/bin/record 'Have I Got News for You s44e10 Christmas Special on BBC One for 31' # Actor Daniel Radcliffe, perhaps still best known to most as boy wizard Harry Potter (despite having laid down his wand 18 months ago) is tonight's host, taking charge of Muggle team captains Ian Hislop and Paul Merton, and panellists Andy Hamilton and Sara Cox.
45 23 25 12 * root /home/iain/bin/record 'Have I Got a Bit More News for You s44e10 Have I Got a Bit More News for You Christmas Special on BBC One for 46' # Actor Daniel Radcliffe, perhaps still best known to most as boy wizard Harry Potter (despite having laid down his wand 18 months ago) is tonight's host, taking charge of Muggle team captains Ian Hislop and Paul Merton, and panellists Andy Hamilton and Sara Cox.
30 19 27 12 * root /home/iain/bin/record 'Have I Got News for You s44e11 Have I Got News for You 2012 on BBC One for 31' # A compilation of highlights from the past year, remembering how Paul Merton, Ian Hislop and a variety of guest hosts and panellists took on the big news stories of 2012. Last in the series.

# Homeland after Season 02 episode 11
00 21 23 12 * root /home/iain/bin/record 'Homeland s2e12 The Choice on Channel 4 for 91' # Feature-length conclusion of the second series of the Emmy award-winning thriller. Carrie thinks about returning to the CIA, but wonders if a career in the intelligence service is really for her, and Nicholas Brody meets with Faber and considers his family's future. Meanwhile, Saul is ordered to undertake a secret mission and Quinn makes a momentous decision. Last in the series.

# The Snowman and The Snowdog
00 20 24 12 * root /home/iain/bin/record 'The Snowman and the Snowdog on Channel 4 for 31' # Animated sequel to Raymond Briggs' classic festive tale The Snowman, telling the story of another youngster's magical Christmas. A boy's snowman and snowdog come to life at the stroke of midnight and take him on an adventure to the North Pole, where he and his new companions meet an assortment of colourful characters, including Santa himself, before returning home - where a wonderful surprise awaits.

# The Royal Institution Christmas Lectures
55 02 27 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e1 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Air: The Elixir of Life on BBC Four for 61' # Peter Wothers explores the scientific elements using a periodic table made from audience members at the Royal Institution in London, to help uncover what the medieval alchemists knew about the air people breathe. In his investigation, he reveals these elements can be used to control fire, defy gravity and harness the power of a lightning storm.
00 20 27 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e2 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Water: The Fountain of Youth on BBC Four for 61' # Dr Peter Wothers investigates whether drinking water can restore his youth, and discovers how exploding balloons could help solve the energy crisis. The presenter is also joined by Paralympic gold medal-winning cyclist Mark Colbourne as they try to find out what happens when two of the most reactive elements on the periodic table, caesium and fluorine, meet.
00 20 28 12 * root /home/iain/bin/record 'The Royal Institution Christmas Lectures s1e3 Royal Institution Christmas Lectures 2012: The Modern Alchemist: Earth: The Philosopher'\''s Stone on BBC Four for 61' # Dr Peter Wothers explores the elements within the earth and investigates whether it is possible to extract the world's most valuable minerals from them. He discovers how carbon dioxide can be turned into diamonds and attempts the ambitious feat of turning lead into gold. Joined by Professor Sir Harry Kroto, the pair find out what happens when you set fire to a diamond, and establish whether a member of the audience is worth their weight in gold. Last in the series.

Adding a comment line with the show name and previous episode makes it reasonably human-readable and tacking on a comment at the end of the cron line allows you to add in the episode synopsis. Works pretty well.

Smoothing the process

This is nice, but unless you have file name OCD, you won’t have a nice neat folder of TV shows. That’s why in the source of some of those scripts there’s reference to show_organise.php. This looks at any video file in a target folder, parses out the name, episode number and subtitle and then moves it into the TV shows folder in a sub-folder with the name of that show. It then writes a symlink to that file back to a “queue” folder so that there is one location to look for unwatched TV shows. Then is looks at the database file for XBMC and checks which of the shows in the queue folder have already been watched and it deletes the symlink for those shows. The queue folder now represents an unattended churn of just shows that haven’t been watched.

No one likes TV ads

Watching recorded TV is great for being able to fast-forward over the ads, but who wants to have to put their drink down and reach for the remote to do that? I found a reasonably reliable solution for skipping UK TV ads on the mythtv wiki. This ignores most ad detection tactics – looking for a network logo in the corner etc and takes advantage of a feature of UK commercial TV: All ad breaks are preceded and ended with tiny block of silence. The example script then takes the audio from a recording and uses mp3split (designed for splitting continuous audio into separate tracks) and uses the position of those splits to determine which are ads and which are the actual TV show. I tweaked this script and included it at the end of my record script so that it outputs edit decision list files which XBMC will automatically pick up and use in order to skip over ad breaks with no user intervention. Sweet, sweet sanity.

Remote schedule

This isn’t quite Sky+ though is it? I mean, with Sky+ you get an app which lets you schedule TV shows to record from your phone. You can however, get pretty close with a bit of Dropbox (or cloud storage service of your choice) trickery.

Step 1: Add your subscriptions file to Dropbox. I’m using the linux CLI client.

Step 2: Symlink from that file back to where the scheduler is looking, e.g. ln -s ~/Dropbox/subscriptions.txt /share/tv/subscriptions

Step 3: Monitor that file for changes. inotify is the swiss army knife of scripting when it comes for monitoring for file changes but it’s less straightforward if you want to set up what is effectively a service running constantly against a file. I did find a neat little script called when-changed, however which simplifies this massively. I just then have it set up so that whenever my subscriptions or listings file is changed, it re-runs the scheduler.

This is the command which I have set up:
/home/iain/bin/when-changed /share/tv/subscriptions /share/tv/listings.xml -c "/home/iain/bin/tv_priority_schedule | tee /etc/cron.d/tv-schedule /home/iain/Dropbox/tv-schedule.copy.txt" &

This checks the subscriptions and listings files for changes and runs the TV scheduler if it finds any. The output is a crontab file which it writes to cron.d but also creates a copy in my Dropbox. Now I can edit my listings in Dropbox on my phone and see the resulting TV schedule update a short while later, also on my phone.

Of course, if I wanted I could also have it automatically compress recordings and have them dropped into my Dropbox as well for remote watching, but I think that’s for a time when I have a fatter upstream…

Quick And Dirty DVR

October 26th, 2011 by Strawp

I recently dug out an old USB 1.1 Digital TV Tuner – a Hauppauge WinTV Nova-T USB, which I think I bought in about 2003 and eventually gave up on due to poor reliability under Windows, the crappy TV signal quality in Coventry and the success of excellent torrent sites like UKNova. Well, I’ve moved house now, and with an increased TV signal strength also came the bad news that I appear to be on a limited bandwidth ADSL line. I envisaged a single evening of plugging in the tuner, installing MythTV under Ubuntu and having a neat DVR to use.

Sadly, this was not to be the case.

Getting the tuner recognised under linux wasn’t too hard. The required firmware was already present in Ubuntu’s repositories but I couldn’t get the thing to scan. MythTV couldn’t open or ID the card and “scan” resulted in nothing. I even tested with the intended packaged drivers under Windows and got about as far.

Eventually, I found “w_scan” which does the kind of full-frequency scan your TV would do and was able to produce a channels.conf file in the format that tzap uses. Success! On the tzap page of the MythTV wiki it shows how you can use tzap to tune the device and “cat” to just dump the MPEG stream to file. Excellent – time for a quick and dirty script!

I then knocked up a “record” script, which takes easy-to-read commands like “record Eastenders on BBC ONE for 30”, tunes the card and dumps the MPEG stream to a sensible location. Combine that with some cron and I’ve got a hacky little DVR. XBMC can do the front end stuff.

I now have to get used to the idea of knowing I want to watch something before it airs, like we used to do in the 90s.

Here’s my TV shows crontab as an example:

# Record soaps off the TV
29  19  * * 2,4 /home/iain/bin/record Eastenders $(date +\%F_\%A) on BBC ONE for 32
59  19  * * 1,5 /home/iain/bin/record Eastenders $(date +\%F_\%A) on BBC ONE for 32
30  18  * * 1-5 /home/iain/bin/record Hollyoaks $(date +\%F_\%A) on Channel 4 for 27

# Watchable stuff
59  20  * * 5 /home/iain/bin/record Have I Got News For You $(date +\%F) on BBC ONE for 32
59  21  * * 5 /home/iain/bin/record QI $(date +\%F) on BBC TWO for 32
0   18  * * 1-5 /home/iain/bin/record The Simpsons $(date +\%F) on Channel 4 for 30

Or for one-offs (as root, unless you change permissions on the device):

echo "record some tv on bbc one for 25" | at 16:00

(or as commandlinefu.com would probably prefer it:)

at 16:00 <<< 'record some tv on bbc one for 25'

Let me know in the comments if you end up using it.

DIY Hands Free Kit

May 10th, 2011 by Strawp

A few years back I came up with the CarPC as a cheapass hacky way of getting MP3 playback integrated into my unspectacular car. Since then, the world of mobile has moved on a whole lot and the laptop stashed in my boot actually pales in comparison with the features (and power) of my Desire Z Android phone. Plus, I’m already syncing music with my phone, like I used to with my car PC and with Google Navigation I have a powerful satnav solution already at hand. All I really needed at this juncture was some kind of hands free setup.

Bluetooth control?

Grom Audio, who made the CD autochanger interface that I used for my car PC also make an autochanger interface which extends with a bluetooth module. This seemed like an unnecessary expense however – I already had the Car2PC autochanger interface which still acts as a high quality stereo input even without the USB control via the car head unit.

The official HTC headset

The headset that came with my phone has an integrated mic with three buttons: answer call/play/pause, next track and previous track. Wouldn’t it be cool if someone had worked out what the circuit inside that thing was? I googled around a while but in the end, one of my friends pointed me to George Smart’s Wiki, where he’d already sacrificed his headset and worked out a circuit diagram. This was massively useful, and given that the circuit consisted of a few cheap components, I thought I’d see

if I could give it a go and make a working copy.

Circuit prototype

If, like me you haven’t done any electronics since you were at school, it’s a good idea before you start soldering to get a prototype board. This is the Lego of electronics – you can stick components into the board without solder and switch things around if needed. I got the smallest prototype board I could find from Maplin for a couple of quid and had a circuit which gave the right resistances fairly quickly. The idea was to produce something I could plug my phone into on one end and would have stereo output and a mic connection so that any generic mic with a 3.5mm plug could be used with it. In addition to the components list on George’s wiki I needed an extra 3.5mm microphone socket, a 3.5mm stereo socket, a 3.5mm 4-pole plug (for connecting to the phone).

Now, it turns out the connecting to the phone part was problematic. The audio connection on HTC phones is pretty much the same as an iPhone. It’s compatible with a standard stereo heaphone connection (3.5mm 3-pole plug) but it has a third “control” ring on the plug which is used to carry the signal for the microphone. Shorting this signal (with the play button) sends the signal to the device to play music and the next/previous buttons simply send approximate voltage signals down the wire by varying the resistance slightly. In order for the controls to work I needed a 4-pole plug. You can get 3.5mm 4-pole plugs from Maplins, but holy crap are they fiddly to solder. I spent a couple of really frustrating hours trying to get four lots of wire to stick to the really unhelpful contact surface on the plug I bought. These were not designed for people whose GCSE in Technology a distant memory.

In the end I gave up and managed to find a much more solder-friendly 4-pole socket on eBay and got a pre-made male-to-male 4-pole plug cable (which you can even pick up in HMV these days).

Two resistances and a microphone

Here’s the part where I’m glad I used a prototype board first. I selected a reasonable looking tie-clip mic online that would handle calls and voice commands for the phone and plugged it into the circuit. Suddenly, the “next track” button didn’t work. What had transpired was that by adding what was effectively a 1.4 kilo-ohm resistor to the circuit in parallel, I’d lowered the effective resistance when I pressed the next track button. This meant the phone interpreted it as “previous track”. To work around this, I needed to raise the resistance on the “next track” button. If you are trying this and get the problem, my method was to measure resistance across the mic with a multimeter, then using the resistors in parallel equation plug in the target resistance on the control wire (560 ohms) and the value of the other parallel resistor on the circuit. With a little help from ElectroDroid, this gave me a new resistor value to plug in (1 kilo-ohm, I think I went for) which then made all the buttons work with the mic attached.

Constructing the device

I guessed an approximate size for the circuitry, buttons and sockets and got a small project box from Maplin. For my first attempt at this, I figured I’d make life easy for myself and just put all the connections on top so tha

t I could work on cutting the holes to the right shape and getting everything to fit in, then I could just pop the circuit board into the box and screw the lid down.

The photo is what I ended up with. From the left, the interface is:

  • Stereo audio out (to the car audio system)
  • 4-pole 3.5mm stereo audio and control I/O (the phone)
  • 3.5mm 2-pole mic socket (goes to tie-clip mic, which is going above the door frame in my car)
  • Previous track button
  • Play/answer button
  • Next track button

Seamless automobile integration AKA “The part with the duct tape”

So that’s all the bits working. Now to find a space in my car that I can fit this where it’s in easy reach, won’t get in the way and can be secured easily. In the end I went for the space just in front of my gear stick, mounting it next to the 12v power socket, over the top of the ash tray. The stereo cable for the car audio actually goes through the back of the car and plugs into the Grom Audio CD autochanger emulator. The mic cable is taped around the dash up the side of the windscreen and sits near the top of the door, in clear audible range. The phone is held in a generic phone mount on the top of the dashboard and plugged in to the 12v power (converted for the phone of course).

I might neaten it up. I probably won’t.

Extending the controls

Having the standard audio controls, plus the ability to take calls is all well and good, but what about being able to do other useful stuff with those 3 buttons? I’ve previously blogged an example of using Tasker with the Headset Button Controller app, so I won’t labour the point, but suffice to say with these two little gems you can pretty much tie or automate any Android task you need to with only 3 buttons at your disposal. I keep my Tasker profile checked into SVN if you want to borrow bits of code, and I’m changing it all the time but here’s some of the things I’ve used this setup for:

  • Press a button to activate voice commands (“navigate to cambridge”, “call home”, “listen to beastie boys” etc)
  • “Love” the currently playing track
  • Automatically start playing music when headset and power is attached (poor man’s dock mode)
  • Automatically read out incoming texts using text-to-speech
  • Announce callers using text-to-speech
  • Re-read the last text message
  • Call a specific person using a button press
  • Launch navigation to a specific place with a button press (i.e. the “oh shit, I’m lost” button)
  • Lock onto heat signatures and engage stinger missiles

So there we go. That’s the kind of car setup that you’re only just starting to see in new cars at the moment (at time of writing, ahem) and for complete pocket money.

“Love” tracks on last.fm with one click of a headset button

April 30th, 2011 by Strawp

Here’s a handy little trick for keen last.fm users on Android that lets you mark the current track as “Loved” with a headset button combination. What you will need is:

  1. The official Last.fm client for Android
  2. Tasker
  3. Headset Button Controller
  4. A headset remote button or the headphones that probably came with your phone if it’s got buttons on it

Set up last.fm

Nothing much to do other than log in and make sure your media player of choice scrobbles tracks correctly. I’m using Winamp which works fine, but make sure you’ve turned on scrobbling in the settings. The last.fm widget has a “love” icon which due to the awesomeness of Android’s intents architecture we will be calling on. Add the widget somewhere on your home screen and make sure that bit works.

Create a “Love Track” task in Tasker

In Tasker, click “Tasks” -> “New Task” (call it “Love Track”) and in the task edit window add “Misc” -> “Action Intent” and in the “Action” setting, enter “fm.last.android.widget.LOVE”. Leave the other fields blank and make sure “Target” is set to “Broadcast Receiver”. You could also add an action which says “You love this track” by using “Misc” -> “Say”.

Start a track playing and come back to the edit screen and then hit “Test”. The last.fm app should create a little toast notification (one line of text overlayed on the screen) saying the track has been marked as played. After a short while it should appear in your last.fm profile.

Optional: Create a home screen shortcut

Long-click on a space on your Android home screen and choose “shortcuts” -> “Task” -> “Love Track”. You can now click on that to mark the currently playing track.

Headset button controller

Headset button controller is a stupendously useful app that allows you to assign actions to headset buttons. If your headset only has one button this is great because you can do more than answer/hang up and play/pause by assigning double, triple, quadruple and long-click actions to a button.

Once you’ve installed this set up your usual button actions, simply set one to “Tasker Task” -> “Love Track” and you’re done!

Remote Wireless Music Syncing Android and Linux

February 13th, 2011 by Strawp

Recently, an Android version of Winamp was released and included in its feature list was very useful wireless syncing with the Windows version of Winamp. Unfortunately, my file server at home doesn’t run Windows and Winamp under Wine is an unstable mess, so attempting to use that feature was out. Like a good Linux user, I didn’t take this lying down – we don’t sit around like Windows users waiting for someone to build the solution for us, we cobble something together using tools already available! What I decided to do was to emulate the mechanism that I came up with when I built my Car PC. Basically, the core of wireless syncing in the Car PC was rsync, combined with a little bit of logic to get a list of music track locations to pull over. By offloading some of that logic onto the server, this seemed possible.

First, I installed an rsync app for Android and tested it out. Rsync backup is just a front end for rsync, which is perfect because it lets you use all the normal rsync options, allows public key authentication (so you never need to enter a password) and displays rsync’s output to a log window. I wanted to pull files from my server so in my rsync profile I selected “rsync in reverse” and added the command line options “-vHrtDL --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --progress --partial” most of which is the default for a new profile, but I added:

  • -L, to follow symlinks
  • --progress, to display file transfer progress
  • --partial, to allow file transfer resuming

Android won’t know any of the new music files have been added yet, so they won’t show up in any music players (which all look up available tracks in the built-in Android media database), so you need to install something like SDrescan to run after the files have copied over which will magically make them visible in the music app of your choice.

That’s the phone side sorted out, I just need something sensible to point it at – I don’t want rsync recursing over my entire music collection wirelessly, it would take ages.

On the server, I’m running the (somewhat ageing) Firefly Media Server, which indexes and serves music for my Roku Soundbridge. I’d previously written a data abstraction class in PHP for the sqlite database it runs on, so I used that to produce a script (to run periodically under cron) which picks a list of albums from a playlist (created by another script) and creates a bunch of symlinks to them in the folder that I set as the target in my rsync profile, the idea being I have a regularly changing folder which one rsync command can look at to pick up new music.

I now have wireless syncing that works anywhere with an internet connection, as well as on the local network (unlike Winamp) and the idea of keeping a static rsync profile and using a source folder full of symlinks provides a cunning way of pushing content onto your phone. It would be quite easy to create something like a web interface to pick files on the server and have symlinks of them created on a single rsync source folder. On top of this, Rsync backup provides hooks for Tasker to link into, so you could set this up to run automatically at some opportune moment.

How FireShepherd could live up to its name

November 3rd, 2010 by Strawp

First there was FireSheep. It allows anyone to hijack HTTP session cookies for a number of sites for anyone using them on the same open access point as you. Now, a predictable counter point for that is that someone would come up with “FireShepherd” to protect this poor flock. However, FireShepherd is no where near as fun as FireSheep – all it does is try and crash FireSheep with fake data and hope for the best, meanwhile your session info is still being transmitted in the clear.

Ideas for FireShepherd to be more useful/fun:

  1. Have it force SSL connections on all the same sites that FireSheep snoops on, making session hijacking impossible. Plugins like Force-TLS do this.
  2. Have it create bogus logins to sites where the user’s profile pic has been set as goatse, tubgirl etc. When the FireSheep user grabs that user’s session data they will have those lovely pics appear in their stolen sessions list.
  3. (getting crazy here) have it perform a man-in-the-middle attack on the wireless network, replacing the network’s router as the default gateway or DNS server. You can then point people to fake versions of captured websites and feed the FireSheep user whatever you want. Oops, there’s goatse again! Oh, what’s that you just went to? A malware site? Careless FireSheep user!

Anyway, there’re some ideas. As Steve Gibson pointed out in the last Security Now, simply switching a network to WPA is enough to protect all the users from this attack. If you’re running a Cafe and want to provide free wifi you can make the network password as public as you want – make a poster and stick it above the till. It is unencrypted wifi, not wifi itself that allows user sessions to be hijacked like this.

Proof of concept: Locating a remote machine using the Google API

October 27th, 2010 by Strawp

example output of the locate script

A couple of months ago Samy Kamkar presented a cool hack at the Black Hat Conference which demonstrated that by using a cunningly constructed URL against specific internet routes, you could inject some javascript on their configuration pages and trick them into sending their own MAC address (the router’s own unique code) to a script which would use it to look up their address on Google’s API, thus telling the attacker a pretty good estimate of your physical location.

This was only possible because as well as photographing everything, the Google Streetview cars have been recording the locations of every wireless access point they encountered. By recording the signal strengths of certain access points by the location they were observed in it’s possible to do a simple triangulation calculation and get a pretty good estimate of where the access point is.

I found this pretty fascinating, so I created a little PHP script to use this trick, perhaps it could be used to help locate a stolen laptop, for example. The script works like this:

  1. Attempt to scan using the wireless network adapter for any nearby access points. Save their key details (BSSID, signal strength and most importantly, the MAC)
  2. Compile all this into POST data and send it to Google using libcurl. Even without nearby MACs Google’s location API will do a better job at locating the machine than the usual GeoIP services.
  3. Get a bunch of data back from Google, including longitude and latitude estimates and a street address. For ease this also gives you a Google Maps link

The accuracy depends on how many neighbours your wireless card picks up and how much data Google has harvested from the wireless networks on your road, but for most people it will be accurate almost to the right house number.

The script is on my SVN server for download