Precomputing for fun and profit

    Last year I wrote a post about my keyword analysis tool. I wrote this piece of software in the years before I started working for OUP, and only now I have seen the tools at the disposal of the lexicographers do I realise that I created what was in effect a corpus analysis tool.
    I used the technologies I was familiar with using for web sites, PHP and MySQL. These are the bread-and-butter of web development, making it possible to create dynamic and flexible web content engines with relative ease. Since an SQL engine is designed for the querying and analysis of data, I thought, MySQL would be perfect for the task of language analysis.
    I was proved right, to a point. I was able to quickly hack together a fully functional tool that would analyse a corpus to provide me with collocates and their frequencies for any n-word phrase. But in sticking dogmatically to what I knew, beyond that point I had taken entirely the wrong approach. I was using a very powerful database engine to perform the same very complex task again and again on an extremely large data set, and as my corpus edged into the tens of millions of words I found that my analysis was redolent of the early days of mainframe computing when programmers would start a job and go away to have a cup of tea while it ran. The system became so slow as to be unusable. It works, but even with all the MySQL optimisations in the book it does not do so in the real time demanded by today's users.
    My mistake was to rely on an expensive commodity when I should have used a cheap one. When I first used computers twenty-plus years ago, storage was the expensive commodity. We used to have disk compression systems using a cheap commodity - the processor time on our PC-ATs - to expand an expensive commodity - the free space on out 10 and 20Mb hard drives. Thus we were trained as programmers to be as efficient as possible in our use of storage space and memory.
    Space efficiency is a good thing if you are programming for a tiny embedded system but in 2011 disk space on web servers is laughably cheap. Processing power, as represented by the waiting time of your users, is not. Clearly a rethink is called for.
    So as an experiment I tried precomputing collocates and frequencies a subset of phrases and saving them as a structure of separate tiny JSON files. A simple JQuery based browser interface was then cobbled together to browse them, resulting in an application that was near-instantaneous for analysis that would have taken minutes to compute all the separate component queries using the MySQL tool.
    The moral of this story is I guess to stand back from your choice of underlying software tools and ask yourself whether they have been driven by suitability or familiarity. I used PHP/MySQL because I am used to it, I never gave a thought to whether a simpler solution might not be much better.
    So I'll be revisiting my hacked-together precomputing engine and applying it to my entire corpus. I may end up with a few gigabytes of JSON files instead of a large MySQL database but to the user that doesn't matter.
    I never thought I'd be returning to flat file storage, in fact I've mocked people who use perl DB files for similar tasks in the past. Guess I'd better eat those words then, precomputed of course!

The annoyance of writing stupid things

   It is with some surprise that I notice six months have passed since I last published anything here. It's not that I haven't written anything, I see I have four draft posts stored up, or that I haven't been writing things elsewhere, just that what I have written here has seemed so banal when reviewed that I haven't pressed that 'Publish' button.

   Better to say nothing than say too much and confirm to the world that you are a fool.

My travels over four decades

  While planning a holiday with my wife a few weeks ago I was left wondering where in the UK I had yet to see. She doesn't hail from these shores, so I wanted to show her something new.
   I started with a map book, but soon realised a broader stroke was required. I can get too engrossed in maps. So I found a blank topographical map of the UK, loaded it into Paint Shop Pro and started work with a virtual marker pen. I had to rely on memory because I haven't kept a travel diary over my lifetime, but over four layers I was able to arrive at a pretty comprehensive itinerary of my life.

(1)The complete picture:

    No prizes for guessing from this map that I've spent my life based in the south Midlands, or that I've spent a lot of time in Yorkshire. Growing up near Oxford and attending Hull University explain those two. You can also trace some of my childhood holidays in Devon, Dorset or South Wales. But the rest is a surprisingly disjointed array of trips to see friends and relatives, plus road trips to seek out interesting places. This map might be a lot less diverse if I didn't own a motorcycle.
    Surprises: I've never been to North Wales, or central Devon. Somewhere to take my wife on holiday, mission accomplished!

(2) The 1970s
Childhood holidays and trips to see my aunts. Memories of sticking to the vinyl seats of an Austin Allegro or British Rail in the Age of the Train.

(3) The 1980s
    Confessions of a teenaged train geek. Four kids with a Family Railcard. Plus yet more family holidays to Yorkshire, this time in an Allegro with velour seats. That's progress, that is.

(4) The 1990s
    Someone gives me a driving licence and a succession of tatty old gas-guzzlers take me further afield. How much petrol did I burn in that decade?

(5) The 2000s
    Finally I'm old enough to afford a nice car and I've got an entire country to show my wife. So how did I manage to miss out North Wales, most of Devon and the Scottish Lowlands?

A dictionary search box for your website or blog

The Oxford Dictionaries Online search box widget I mentioned in a previous post has now gone live and is available for you to put on your own blogs or web sites. It is available in two versions, one for each of world and US English, and an instance of the final result can be seen at the top of the right hand column of this blog.

Get it here: http://oxforddictionaries.com/page/searchbox

Yet another set of browser stats

    It's worth posting these graphs of browser market share over the second half of 2010 generated from real-world Urchin traffic figures. The first one shows the percentage traffic for each of the five main browsers, and the second one breaks the MS Internet Explorer figure down into separate versions since version 6. April and May are best ignored due to the site launch distorting the figures, but June onwards can be considered to be an accurate representation of Internet users.

Web traffic by browser

    As MSIE has slipped gently below a 50% share, Firefox has remained essentially steady, leaving Chrome to pick up IE's defectors. The slight surprise in this graph is at the bottom, Safari has been overtaken by Opera in the later months of 2010.

MSIE individual versions share of MSIE total

    As might be expected, MSIE8 has taken over from MSIE7, which is slowly levelling off. Whether MSIE7 will follow MSIE6 in refusing to die remains to be seen, but the continued use of these two browsers does indicate that a substantial core of Windows users are refusing to upgrade either manually or by Windows Update. MSIE6 shows a slight increase in the summer months, this is probably due to students using home PCs while away from their schools. MSIE9 has a very small showing, as you might expect from its pre-release status.

2010 as viewed through Daily Express headlines

    A couple of years ago I was working in a small business centre in Oxfordshire, a building that was home to several small tech companies. We had a common room with a kitchen and dining area, and the building management provided us with a selection of daily newspapers.
    Among those papers were a high-end broadsheet, a red-top tabloid and the Daily Express. I became fascinated by the Express headlines over the year, and resolved to collect them, initially just for the ones barmy enough to make me laugh but later all of them for a collective analysis. This post is the result, an infographic that tries to capture the essence of a year as seen on the front pages of the Express.
    So why did the Express fascinate me? Here is a gloriously barmy  newspaper obsessed with cancer stories, lurid tales of illegal immigrants, and the UK housing market, using a colourful vocabulary all of its own to stir its readers into righteous anger over stories that very often have little relationship to the main news stories of the day. Everyone is furious or outraged in Express-land, we're under siege from miscellaneous foreigners and there is no Government decision or economic movement that is not designed to let down the Express reader. This continual negativity is offset with ludicrous non-stories proclaiming miraculous rises in the housing market or turning tentative conclusions in scientific papers into medical breakthroughs in the fight against cancer. Seeing what they had on the front page became part of the entertainment of my morning routine, there is nothing like a ludicrous headline over your morning coffee for starting the day with a laugh.
    The infographic shows some of the pretty pictures that can be gleaned from a year's Express headlines. A PNG version can be downloaded by clicking on the image or you can download it as a PDF from this Google Docs link. I hope you enjoy reading it as much as I enjoyed making it.

Three easy predictions for 2011

   ...All of which will look delightfully naïve by late December 2011.

(1) People will consider Microsoft Windows Phones worth having
    Windows-toting mobile phones have never been cool, have they. Everyone's got at least one friend who's got one, but they're always the friend that has to enthuse a little too much about their mobile because it doesn't quite deliver. I remember being quite excited about one friend's XDA, before finding just how clunky it was compared to the ageing Nokia running Symbian I had at the time. A high end smartphone has to be *really* clunky to make the Symbian interface from circa 2005 seem slick and easy to use.
    Windows Phone 7 has changed all that in my view. Sure it's no iPhone, but the interface doesn't make you want to smash the phone in frustration and unlike some earlier Windows Phone offerings it delivers what you expect of it. But that's not the key feature that I think will make Windows Phone 7 cool in 2011.
    The first Windows Mobile phone I ever saw was owned by a friend who is a .net developer. He was excitedly showing his desktop .net  app of the moment running on his phone. It was pretty useless with a tiny screen, but it ran. Because as a Microsoft device it came bundled with the .net runtime, and as a .net developer his skills were instantly transferrable to his phone.
    And there is the reason I think Windows Phone 7  phones will become cool. Apps, and its tight integration of .net. Decent hardware paired with a lot of marketing and a good app store, these phones will sell. And since there is a huge developer community of .net developers who do not need to learn any new skills to create Windows Phone 7 apps, I think that Microsoft will not face the same challenges that Symbian or Bada face to attract developers in 2011, and the resulting high quality apps will make Windows Phones worth having. Not even Microsoft could screw up such an opportunity, could they?

(2) Companies and the media will wake up to the idea that not just the iPhone can run apps
    Am I the only person who is sick of seeing companies, newspapers and others proudly touting their new mobile capabilities with an iPhone app? My prediction for 2011: a chill wind will blow through the marketing departments of the land as they realise that all the cash they spent on that shiny new app isn't going to make them a penny from all the millions of owners of Android and other smartphones who can't run iPhone apps.
    Whether this means that they will start offering a range of apps for different platforms or simply move their app content to in-browser online delivery I wouldn't dare to predict. I'd expect the latter though. Shame it's a little early for W3C Widgets.

(3)Tablet computers will become cheap enough to be given away free
    If you are unfortunate enough to have handled one of the cheap and nasty Android tablets that have appeared in the last few months, you'll know just how nasty they are. Horrible user interfaces on resistive screens, too-slow processors, not enough memory and too-short battery lives. At around about £100, they are a huge disappointment and waste of money.
    But how would you view them if they sold for £50? Too much? How about £25? Or how about free? Imagine, you are a large home-delivery or mail-order business, perhaps a supermarket. You have millions of customers and you want them to be able to access your service with the minimum fuss wherever they are. If only they could just pick up a tablent while they are slumped in front of the telly, go straight to your site and spend their money.
    If you are a nationwide supermarket chain, you have the buying power to pick up cheap and nasty Android tablets from China in shipments of millions for relative pennies per unit. And if by giving away a tablet to each customer you guarantee yourself a massive sales increase, then a free tablet for all becomes a no-brainer.
    I don't know who will be first to do it. Tesco perhaps, to their phone or broadband subscribers. But I'd be surprised if 2011 didn't see at least one free branded tablet in the living rooms of the UK.

    So there you go. I've set myself up for a fall this time next year, but I think these three really do have a chance of happening in 2011. Only sitting out the year can tell me the really interesting stuff I missed.

Worthless by degree

    In the wake of the recent student protests over tuition fees, I have heard more than one conversation recently on the subject of students, their resources and the courses they are taking at university. Most of these conversations have passed me by, there was little point in my participating as the scale of the ensuing argument would be out of proportion to the scale of the points I might make.
    One friend of mine annoyed me enough to pull out my keyboard though, with the assertion that "worthwhile" courses like science and engineering should be fully funded while "worthless" degrees such as art history or media studies should receive no funding. The speaker expected my agreement, as my degree is in electronic engineering, and was quite surprised when I rather vehemently expressed opposition.
    You might expect that I'd be about to say that all degrees are worthwhile, and that art historians and media students deserve support as much as electronic engineers. And I won't disappoint you there, after all my sister is an art historian who spends a lot of her time ensuring that media studies students have at least one course they have to put in some serious work at to pass.
    But what you might not expect is that I would attack the notion that an engineering degree is a worthwhile degree that should be viewed as more important for funding than any other. My experience of having an engineering degree is that once outside the walls of the university it really hasn't been of much use to me directly other than as a bit of paper, the teaching was designed to breed engineering PhD students rather than equip us for life in industry and everything I have done for a living since has either been learned on the job or based upon something I learned outside university. Yes, my whole career as a web developer started with Microsoft ASP back in the 1990s, which being a BASIC scripting language was a skill I built on something I learned as a spotty kid long before I reached university.
    "But the country needs engineers!" I hear you say. And I'd have to agree with you there. Though I think the meaning most people would ascribe to that sentence is rather different to the meaning I would ascribe to it.
    To me, the country needs engineers rather than simply people with engineering degrees at all costs. In other words, the country needs as its engineers people who are born to engineering as a vocation, a gift even. It's never been viewed as something particularly special but it should be, I doubt I'll hear many middle-class parents enthusing about their offspring's aptitude for Meccano or coding in the same way they would if their little Tarquin turned out to be a musical prodigy. And just as there are born engineers, so are there born art historians and even born media students who would be utterly wasted on an engineering course.
    If we have created a world in which tertiary education is near-mandatory and a degree certificate is required to embark on careers for which a school certificate would have been required for our grandparents' generation (and yes, I count my career among those), it seems to me crazy to force people into degree courses purely for ideological reasons that they are never going to find any value from and which will have little bearing on their eventual careers.
    Far better to have degree courses on which places are earned by merit and whose subjects depend on the aptitude of the student. Oh wait, wasn't that what we had twenty years ago?

Strabismus

   This morning I stumbled on a site that made me think a little. It's a site promoting an ebook by the parents of a child with a strabismus, detailing their child's treatment. I left the following comment.

I have to admit to reading this site with mixed emotions. I’m a 40 year old British person with a significant strabismus, something I’ve had since birth. I don’t have hardware 3D vision and never will have, my eyes are like separate cameras meaning I can switch between two views of the world.
This has never been a problem for me, it has only very ocasionally been a problem for others. It has never held me back in life, education or the workplace. So I can never be a fighter pilot, but does that matter?
I understand why you were horrified by it in your son but speaking personally I am very glad my parents did not put me through the pain and risk of surgery when I was too young to understand what was going on. I’ve always seen it as very much a part of me and have no desire to seek any treatment for it as an adult.
So those are the effects of strabismus on someone who’s lived it. If one of my kids is born with it I’ll leave it until they are old enough to understand before I ask them if they want surgery.

    It's probably a bit off-message for them so it may not pass their moderation, but I felt it needed to be said. I understand the pain of a parent whose child doesn't meet their idea of perfection, but as one of those imperfect children I squint and I'm proud!

Report from OpenMIC9

     So I spent yesterday at the Jam Factory in the company of about fifty geeks talking about everything mobile at Open Mobile Innovation Camp 9. These type of events are a good place to meet others of like mind and learn about emerging trends and technologies in a particular field. Sometimes the geek level is higher than others and at times one can be left behind by people far further ahead of the wave than oneself, but thankfully yesterday was very accessible and I came away with far more information than I went in with.
    As is often the case at such events, the morning's schedule was filled with talks from selected luminaries of mobile development and the afternoon to "barcamp" sessions - geeks talking shop.
    Google's Reto Meier couldn't make it due to the snow, so we had a talk on the future of mobile technology from IBM research's Dale Lane, a talk on the comemrce and marketing of the mobile app space from Everything Everywhere's Mark Watts-Jones, a talk on upcoming W3C mobile standards including my interest of the day, W3C widgets from Vodafone's internet evangelist Dan Appelquist and a demo of mobile app technologies from Calvium's Tom Melamed.
     In the afternoon's melee I ended up caught up in a weighty discussion on identifying your mobile users and their device capabilities - important to us if we ever want to get the best experience over to our visitors - and then an examination of the evolving nature of a smartphone. Interesting, if rather geeky stuff.
    All in all a worthwhile day. I came away knowing more about widgets which was my objective for the day. I'd like to think in a year's time when I'm waving a phone with a dictionary widget on it at anyone who'll listen, that the marketplace will be just about ready for them and we'll thus steal a march on all our competitors who've rushed headlong into the app stores.
   I can hope, can't I.