The day I was DoSed by Google

    I launched my startup this week. On a very modest scale, no debauched parties or penthouse offices on scads of VC money, just me and my laptop in my dad's living room, popping open a bottle of home made cider to mark the event.

    Language Spy is the tangible fruit of a seven or eight year side project, creating a searchable corpus of political language. It's driven by a pair of Raspberry Pis doing the numbercrunching and uploading data to Google Cloud Storage buckets from whence the site is served by Google App Engine.
    You can see events unfolding through the words used about them, for example the correlation between "Hillary Clinton" and "Email" in the last week of US politics. If like me you're a news junkie, it's compelling viewing.

    Unfortunately though if you click the link as I write this, you'll see a Google App Engine quota exceeded message. The site won't work, because I have reached the point at which I can't afford the traffic it's serving and it has exceeded my daily budget.

    This traffic spike would be no problem if it were generated by real site users as then I'd be able to monetise the traffic, but sadly it isn't. Instead it's generated by GoogleBot. That's right, being indexed by a search engine has taken my site down. The bot looks at the site, decides it's on some very fast infrastructure, and issues millions of requests per hour.

    When I examine my problem, it becomes clear that it has several aspects:

1. It's a language analysis site, so it has a *lot* of pages for the spider to crawl.
2. Being a language analysis site there are no pieces of language I can exclude using robots.txt, so I can't reduce the load by conventional means. How do you decide which language is more important than other pieces? You can't, at least not when your aim is to have it all open for analysis.
3. I can't tell Google to slow down a little, as where I'd expect to be able to do this in Webmaster Tools I get a "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate." message. I see this as the sticking point, if I could restrict Googlebot's rate I'd be able to keep the site running and take the hit of not being so well indexed.

    This means that the spider is eating through my daily Google App Engine quota very quickly indeed. I will find myself gaining a hundred instance hours in a very short time indeed as GAE spins up loads of new instances to deal with the spider. Pretty soon the site hits its daily quota and goes down. I could keep it going by feeding in more money, but I'd have to put hundreds of dollars a day into it and with no end in sight I am not made of money.
    Right now my only hope lies with a crawl issue report I filed with the Webmaster Tools team, if they can give me control over my indexing rate I'll be good to go.  But I can't say when they'll come back to me if ever, so I may just have to come up with a Plan B.
    Is there a moral to this story? Perhaps it's a cautionary tale for a small startup tempted to use cloud hosting. Google Cloud Storage has proved very cost-effective for a huge language database, but the sting in the tail has turned out to be how GoogleBot behaves when it sees a cloud server and how per-instance billing on App Engine handles unexpected traffic surges. The fact that it's Google who are causing me to use up my budget with Google is annoying but not sinister, however neither giving me the option to limit my GAE instance count nor slow down the crawl rate doesn't leave me as the happiest of customers.

    So yes, I've launched a startup. It's live for an hour or two a day while it has budget, in the morning UK time. Perhaps that will be my epitaph.

Small computer, big data

    This post comes as one of the longest running scripts I've ever created has just finished its work. In the last week of January I set my Raspberry Pi to the task of processing 5 years of news stories into a 20Gb tree of JSON files, and here in the second week of March it's completed the task.
    Given that a PC has done the same job in a couple of days the first question anyone would ask is simply this: Why?
    My Pi runs all day every day, 24/7. It collects the news stories from RSS feeds and stores them in a MySQL database. It uses somewhere under 2 watts, and it will do this no matter what I ask it to do because it's plugged in all the time. I can touch its processor with my finger, it's not hot enough to hurt me. My laptop by comparison with its multi-core Intel processor, board full of support chips, and SATA hard disk, uses somewhere under a hundred watts. I can feel the hot air as its fan struggles to shift the heat from the heatsink. I wouldn't like to hold my finger on its processor, assuming I could get past its heat pipe.
    Thus since I'm in no hurry for the data processing it will use a lot less power and it makes more sense for me to run the script on the Pi. This isn't an exercise in using a Pi for the sake of it, instead the Pi was the most appropriate machine for the task.
    So having run a mammoth script on a tiny computer for a couple of months, how did I do it and what did I learn?
    The first thing I'd like to say is that I'm newly impressed with the robustness of Linux. I've run Linux web servers since the 1990s but I've never hammered any of my Linux boxes in quite this way. Despite stealing most of the Pi's memory and processor power with my script it kept on with its everyday tasks, fetching news stories and storing them as always. I could use its web server - a little slowly it's true -, I could use its Samba share and I could keep an eye on its MySQL server. Being impressed with this might seem odd, but I'm more used to hammering a Windows laptop in this way. I know from experience the Windows box has not been so forgiving running earlier iterations of the same script.
    If anybody else fancies hammering their Pi with a couple of months of big data, here's how I did it. The script itself was written in PHP and called from a shell within an instance of screen. This way I could connect and disconnect at will via ssh without stopping the script running. The data came from the MySQL server and was processed to a 64Gb USB Flash disk. The Flash is formatted as ext4 without journaling, this was judged to be the best combination of speed and size efficiency. An early test with a different FAT formatted drive provided a vivid demonstration of filesystem efficiency as the FAT ended up using 80% of the space after only a short period of processing.
    The bottleneck turned out to be the Flash drive, a Lexar JumpDrive. Reading and writing seems to happen in bursts, the script would run quickly for about 30s and then very slowly for the next 30s purely due to disk i/o. In future I might try the same task with USB-to-SATA hard disk, though I'd lose my power advantage.
    So would I do the same again, and how might I change my approach? I think the Pi was a success in terms of reliable unattended operation and in terms of low power usage on a machine I'd have had running anyway. But in terms of data processing efficiency it could have been a lot better. A faster disk and a faster computer - perhaps something with the Pi's power advantage but a bit more processor grunt such as the CubieBoard - would have delivered the goods more quickly for not a huge extra investment. And the operating system though reliable could probably have been improved. I used a stock Raspbian, albeit with the memory allocation for graphics reduced as low as it would go. Perhaps if I'd built an Arch image with a minimum of dross I would have seen a performance increase.
    I used a Raspberry Pi for this job because it was convenient to do so, it uses very little power and I had one that would have been powered up anyway throughout the period the script was running. The Raspberry Pi performed as well as I expected, but I can not conclude anything other than that it is not the ideal computer for this particular job. It is sometimes tempting when you are an enthusiast for a particular platform to see it as ideal for all applications, well in this case that would be folly.
    The Pi will continue to crunch the data it collects, though on a day-to-day basis as part of the collection process. In that it'll be much more suited to the task, as a cron job running in the middle of the night the extra work of a day's keyword crunching won't be noticed. And there's the value in this exercise, something that used to require a PC, a while of my time and a little bit of code has been turned into an automated process running on a £25 computer using negligible power. I think I call that a result, don't you?

What I'm going to do with my Raspberry Pi

    That magic email from Farnell came on Saturday, my Raspberry Pi is in the post!
    So, what am I going to do with it?
    Like loads of other geeks I expect I'll plug it into my telly, connect it to my router and use it as a web terminal and media centre with geek bragging rights. There it'll sit for however many years it takes until I get a new telly or a Raspberry Pi 2, unseen and uncomplaining. But rather wasted, don't you think.

    Somewhere, the Flying Spaghetti Monster has just killed a kitten.

    So what do I really want it to do? I will have a very small and moderately powerful computer - insanely powerful by the standards of a few years ago -  that uses negligible electrical power and can be left on all the time. I'm still going to plug it into my router and telly, but to make it earn its keep I'm going to have it run my keyword analysis tool.
    Events have moved on a little since my blog post describing the tool, but the principle is still the same. I take new posts every day from a big list of RSS feeds and process them for keyword phrases which I store in a database. I can then extract frequencies and collocates over time, which gives me a picture of the interrelationship between the language and terms in the news over any given period. It not only fulfils my original aim of having a tool that would generate keywords and phrases for previously unseen search terms, but also allows any newsworthy subject to be examined in a way that is not possible by any other means.
    The original tool runs in PHP on my Windows laptop. Its MySQL database is pushed well beyond its limit, and I have been working on a version that uses a large directory tree of precomputed JSON files instead. It's an approach I've since also used in my work, relying on the principle that disk space is cheap and quick while complex joins on monster MySQL databases are expensive and slow.
    I could of course compile PHP for my Pi. It's probably already available precompiled anyway. But the Pi's a Python platform (Try saying that after five pints of real cider!) and that offers me a unique opportunity. My PHP code does the job, but it relies on my own language processing libraries which I built myself as a search engine specialist. I'm not a computational linguist so I'd be the first to say that they aren't as good as they could be.
    Python has the incredibly useful Natural Language Toolkit libraries which allow me to do so much more with my source texts, and so much more quickly than my PHP code. So my first effort with my Pi will be to port my keyword tool to Python, using the NLTK instead of my own library. The Pi will still sit behind my telly and be used for the occasional bit of web surfing, but for the rest of the time it'll be crunching keywords and giving me lots of lovely language data that I can work with in real time rather than with enough time to make a cup of tea every time I make a MySQL query.
    In a way I'm not taking advantage of everything the Pi can do. Almost any internet connected computer could do this job, I'm only using a Pi because it's cheap and low power, and I've lusted for one ever since I read their early press releases. Other people will use the Pi's hardware capabilities to do much more eye-catching things. But my Pi, quietly crunching words all day and night behind my telly, will still be earning its keep. It will allow me to learn new things and since its data is likely to end up in some of my work stuff it may even in its own small way make a contribution to the wider understanding of language.
    So that's what I'll be doing with my Pi, what'll you be doing with yours?

Precomputing for fun and profit

    Last year I wrote a post about my keyword analysis tool. I wrote this piece of software in the years before I started working for OUP, and only now I have seen the tools at the disposal of the lexicographers do I realise that I created what was in effect a corpus analysis tool.
    I used the technologies I was familiar with using for web sites, PHP and MySQL. These are the bread-and-butter of web development, making it possible to create dynamic and flexible web content engines with relative ease. Since an SQL engine is designed for the querying and analysis of data, I thought, MySQL would be perfect for the task of language analysis.
    I was proved right, to a point. I was able to quickly hack together a fully functional tool that would analyse a corpus to provide me with collocates and their frequencies for any n-word phrase. But in sticking dogmatically to what I knew, beyond that point I had taken entirely the wrong approach. I was using a very powerful database engine to perform the same very complex task again and again on an extremely large data set, and as my corpus edged into the tens of millions of words I found that my analysis was redolent of the early days of mainframe computing when programmers would start a job and go away to have a cup of tea while it ran. The system became so slow as to be unusable. It works, but even with all the MySQL optimisations in the book it does not do so in the real time demanded by today's users.
    My mistake was to rely on an expensive commodity when I should have used a cheap one. When I first used computers twenty-plus years ago, storage was the expensive commodity. We used to have disk compression systems using a cheap commodity - the processor time on our PC-ATs - to expand an expensive commodity - the free space on out 10 and 20Mb hard drives. Thus we were trained as programmers to be as efficient as possible in our use of storage space and memory.
    Space efficiency is a good thing if you are programming for a tiny embedded system but in 2011 disk space on web servers is laughably cheap. Processing power, as represented by the waiting time of your users, is not. Clearly a rethink is called for.
    So as an experiment I tried precomputing collocates and frequencies a subset of phrases and saving them as a structure of separate tiny JSON files. A simple JQuery based browser interface was then cobbled together to browse them, resulting in an application that was near-instantaneous for analysis that would have taken minutes to compute all the separate component queries using the MySQL tool.
    The moral of this story is I guess to stand back from your choice of underlying software tools and ask yourself whether they have been driven by suitability or familiarity. I used PHP/MySQL because I am used to it, I never gave a thought to whether a simpler solution might not be much better.
    So I'll be revisiting my hacked-together precomputing engine and applying it to my entire corpus. I may end up with a few gigabytes of JSON files instead of a large MySQL database but to the user that doesn't matter.
    I never thought I'd be returning to flat file storage, in fact I've mocked people who use perl DB files for similar tasks in the past. Guess I'd better eat those words then, precomputed of course!

2010 as viewed through Daily Express headlines

    A couple of years ago I was working in a small business centre in Oxfordshire, a building that was home to several small tech companies. We had a common room with a kitchen and dining area, and the building management provided us with a selection of daily newspapers.
    Among those papers were a high-end broadsheet, a red-top tabloid and the Daily Express. I became fascinated by the Express headlines over the year, and resolved to collect them, initially just for the ones barmy enough to make me laugh but later all of them for a collective analysis. This post is the result, an infographic that tries to capture the essence of a year as seen on the front pages of the Express.
    So why did the Express fascinate me? Here is a gloriously barmy  newspaper obsessed with cancer stories, lurid tales of illegal immigrants, and the UK housing market, using a colourful vocabulary all of its own to stir its readers into righteous anger over stories that very often have little relationship to the main news stories of the day. Everyone is furious or outraged in Express-land, we're under siege from miscellaneous foreigners and there is no Government decision or economic movement that is not designed to let down the Express reader. This continual negativity is offset with ludicrous non-stories proclaiming miraculous rises in the housing market or turning tentative conclusions in scientific papers into medical breakthroughs in the fight against cancer. Seeing what they had on the front page became part of the entertainment of my morning routine, there is nothing like a ludicrous headline over your morning coffee for starting the day with a laugh.
    The infographic shows some of the pretty pictures that can be gleaned from a year's Express headlines. A PNG version can be downloaded by clicking on the image or you can download it as a PDF from this Google Docs link. I hope you enjoy reading it as much as I enjoyed making it.

RSS feed keyword analysis for the fun of it

    What do you do when the recession hits and you are made redundant?

    When it happened to me last year, I wrote an RSS feed keyword trend analyser in my new-found free time. Over a year and several million keywords and phrases later I can find associated keywords and phrases and plot graphs for almost anything that's been in the UK mainstream news. Like this one, showing the fortunes of three Labour party leaders over the past few weeks.

   You can clearly see Tony Blair's book launch as the blue hump in the middle, and Ed Miliband's election as party leader in green on the right. Meanwhile Gordon Brown bumps along in the obscurity of his Scottish constituency as the red line. Funny that, the colours were allocated at random by my graphing library yet Blair got the Tory blue.

    As a search engine marketeers tool it's of limited use unless you really are looking at up-to-the-minute trends for very fast moving content. But as a toy, or for finding collocated words and phrases for newsworthy themes, it's shaping up pretty well.

    I'll be dipping back in to this particular well of words again on here from time to time, both from the tech side and just for the joy of playing with some words.