Preparing for MSIE Overtaking Day

    It's been a trope of the web developer's existence for the last decade: Microsoft Internet Explorer won the browser wars in the 1990s, all other browsers are irrelevant. The customer has this firmly lodged in their heads from the days when MSIE had over 90% of the market and demands support for IE in all its forms over support for any other browser. We may just about have won the war over abandoning IE6 support, but we are still demanded to code in all the workarounds demanded by its just-as-creaky younger siblings.
    It might have been true in the mid-2000s to manipulate the famous phrase about IBM to "nobody ever got fired for supporting MSIE". But as a quick look at StatCounter's Global Stats will tell you, over the last couple of years Google Chrome has come from nowhere and is rapidly converging on MSIE's market share. Extrapolate the graph forward a few months, and it becomes obvious that sometime this summer, probably in June, Chrome will overtake MSIE as the world's most popular browser.

    That's right, June 2012 will see MSIE Overtaking Day.

    You might think it would be unwise to break out the champagne though. After all, the top spot is just passing from one big company to another, won't it just be a case of "Here's the new boss, same as the old boss"? In that we're fortunate: unlike MSIE with its proprietary approach to rendering HTML, Chrome is a Webkit browser, underpinned by open source and web standards. So if we code to those standards we can expect it to work without too many tweaks on all browsers that support them.

    No more browser-specific stylesheets, no more special Javascript hacks, no more compatibility libraries.

    But the title of this piece is "Preparing for MSIE Overtaking Day". We're already there, as developers we're used to coding web standards, all our sites already work in Chrome, Firefox, Safari and Opera. It is the non-technical people who need preparing, all those marketing people at the customer, the legal people and the developer project managers who are stuck in that 2000-era trope. I was shocked not too long ago to encounter a site whose contract specified individual browser versions. Not even "version X and above", so when a legitimate bug was reported in a recent browser the response came back that it wasn't supported because the browser wasn't several years old. That kind of thinking is simply not acceptable.
    So we have to think away from the browser in the post-MSIE world of frequently released standards-compliant browsers. We have to sell web standards such as HTM5 rather than support for particular browsers to the non-technical people we encounter as web developers, and we have to hammer home that message using the clearly visible statistics.
    Otherwise we'll still be coding for MSIE7 in 2017 just like some of us had to support MSIE6 in 2010 And that just ain't funny, not at all.

Accessibility: it's an engineering problem

    A  little over a week ago, the British paralympian athlete Tanni Grey-Thompson gave an interview in which she decried the state of accessibility for people with disabilities in the UK and described the experience of having to crawl off a train because the rail employees who had been booked to provide her with a ramp had failed to materialise.
    It has been interesting to watch the commentary unfold over the intervening days. On one hand we have seen many other stories from people with disabilities of being denied the use of services, of being stranded or of being forced to great inconvenience to gain access to things which should be as easy for them as for anyone else. Meanwhile there has been a chorus of discontent from the kind of people who read the Daily Mail, who see the disabled as a fantastically privileged minority who have vast amounts of hard-working able-bodied people's money squandered on them and should accept their lot and stop whinging. To paraphrase one of their battle cries: "If being disabled is so good, why don't you go and live the dream!".
    It's slightly uncomfortable to realise that while the experiences of the people with disabilities are unfortunately too real, there is also a germ of truth in the root of the bitterness from Mail readers. As a country, we have spent a vast amount of money over the last couple of decades on improving accessibility, so where have we failed?
    An exercise I would counsel anyone able-bodied to try is to accompany a wheelchair user across a British city. A simple walk becomes a lengthy traverse; it is sobering to realise how many wheelchair obstacles you pass on foot without realising it.
    Meanwhile the world is festooned with dubious paraphernalia with the aim of improving accessibility. Cash machines have been moved closer to the ground without a thought being given as to whether they need redesigning for use from a perspective of someone with limited mobility or reach. Small business premises sport unused ramps, electric lifts, and disabled toilets used only as dusty storage rooms.
    One is left with the feeling that a vast exercise in box-ticking has been completed. Everywhere has 'done' accessibility because they have the ramps, signs, and lifts to prove it, yet real-world accessibility for people with disabilities remains as elusive as ever.
    I look at this and see not an accessibility problem but an engineering problem. Of course Tanni Grey-Thompson was let down by the rail company, but was the real failure not in the equipment she was provided with? It seems inconceivable that in an age in which we can create robots that can travel to Mars and operate autonomously for years at a time, yet we can't design a lightweight personal mobility aid for someone in a wheelchair that is capable of letting them traverse the kind of step you might encounter between a train and a platform or anywhere else. The Mars robots definitely are rocket science, the mobility aid definitely isn't! Or it shouldn't be, anyway.
    Of course, we engineers have a major failing, we see everything as an engineering problem, we live in a black-and-white world. Engineering can't always fix social or political problems. But engineering is nothing if it isn't the art of making machines to solve physical problems in the real world, and rather evidently the engineering available to people with disabilities isn't fit for purpose. By my observation the basic design of a wheelchair hasn't changed in many decades, is this really a technology that's reached its zenith?
    Perhaps if our Government had spent less time and money patting itself on the back for a successful but ultimately useless box-ticking exercise and instead invested in research and NHS funding for mobility aids that allowed people with disabilities to render some of their accessibility problems irrelevant, there would be no need for any of this. People with disabilities would be able to go where they wanted and get on with their lives just like anyone else.
   But why on earth would any politician want to do that, it wouldn't win any votes from Mail readers, would it!

How the Pi could have been (some of) ours

  (edit: My Pi arrived on 2012-05-14. If you would like to read about my plans for it you can do so here, and my review of it can be found here.)

    So, the Pi will not be ours. At least until April, according to my email from Farnell.
    The Pi? The Raspberry Pi, that is, powerful yet inexpensive single board computer and object of desire. Released to a storm of interest that created a Slashdot-like denial of service to the websites of its two suppliers, its launch left a lot of hopeful would-be buyers disappointed and venting their anger online as the first production run sold out in seconds.
    Now the dust has settled, time for a look at the launch from a customer's perspective.
    I don't think many people involved will disagree with me when I say that the launch of the Raspberry Pi could have gone better. That's not having a go at the Raspberry Pi team, it's simply stating the obvious given hindsight. The Raspberry Pi foundation are a small charitable endeavour and what they have done is amazing, creating their product from nothing and with minimal resources. They are not a huge multinational company with a sales and marketing operation to match so it is unfair to expect them to be able to emulate one. The fact that we'll be able to buy our Pi at all is an incredible achievement, even if we all have to wait a couple of months.
    But it's worth examining the launch from a customer perspective, to quantify what seemed to fail and arrive at some possible solutions. This isn't a "How I would have launched the Raspberry Pi differently from those losers!" piece but a "Gosh, how can I learn from that and what would I do if that happened to my next product launch?" piece.
    So, in the words of an F1 commentator of yore interviewing Johnny Herbert: what went wrong? Here are the answers to that question from my perspective:

• The launch was massively oversubscribed. A hundred thousand geeks were chasing ten thousand boards. Most of these potential customers were always going to be disappointed.
• The launch was at a very odd time of day. A hundred thousand geeks had to get out of bed for 6am. Thus not only were the customers disappointed, they were tired and disappointed.
• The two suppliers - Farnell and RS - completely dropped the ball. In the age of turnkey cloud computing if you know a hundred thousand people are going to come to you all at once for a single page it is not beyond an organisation of their size to direct them to a web presence that can handle that level of traffic. They failed massively, and they will have paid dearly for that failure in lost business while their sites were out of action.
• The email notification failed. For which the Raspberry Pi people apologised, it seemed their email server wasn't up to the volume required. It's a little unfair to put this here because Twitter and the Raspberry Pi blog seemed to do just as good a job, however it was part of the picture that was missing.

     It's easy to get irate about this catalogue of unfulfilled expectations. It is however worth reminding any reader tempted to cry universal failure that the Raspberry Pi people succeeded in doing exactly what they set out to do, which was launch their product and sell their first batch of boards. They were very clear about the size of that first batch beforehand, also they were very clear why they only had that number.
     So, they succeeded in their primary aim, but received an online slamming from disappointed would-be customers. Where did they not succeed, and how might other product launch teams learn from their launch?

• They didn't manage the expectations of their customers. Sure, we all knew it would be busy, but everyone went in thinking that they might have a chance of snagging a Pi. Perhaps a lottery the week before launch to allocate the right to purchase what boards would be available might have contained those expectations.
• The time picked for the launch was in my view unwise. Was it to synchronise with US time zones perhaps, or was it at the behest of RS and Farnell? Either way, it had the effect of intensifying the disappointment of the customers whose expectations had been dashed, not only had they failed to score a Pi but they'd had no reward for getting out of bed early. Yes that sounds petty, but customers are fickle and the best way to get them reaching for a credit card rather than a whiny social media post if you've got them out of bed early is not to annoy them with no reward.
• The email was not farmed out to a server capable of handling the volume of traffic. Yet again this is slightly unfair. Other media did the same job, and their budget is better spent on making more boards than supporting commercial email providers. However when looking at how a commercial product launch could learn from the Pi launch, this is a valid point to consider.

    There is one way in which the episode can be rated as a complete failure though: the server outages from both RS and Farnell. I'm sure the Raspberry Pi team would have done their best to communicate the likely traffic levels to these two suppliers so I can only assume that they did not listen. Or perhaps they did not believe that a small organisation with ten thousand boards to sell could generate that level of interest. Either way the server outage demonstrated the deficiencies of their infrastructure only too well. A right-to-buy lottery would have mitigated the traffic surge, but even without that the suppliers could have set up cloud-hosted Raspberry Pi sales microsites able to handle the traffic. I'm guessing that lost general sales due to the website outages will have focused their minds, and next time it would be different.
    The Raspberry Pi will be an astounding success. Deservedly so, it is an amazing product. And a few whiny geeks on its first day won't change that in the slightest. I guess this piece is looking at the Pi as a case study for more mundane product launches, ones that don't benefit from the goodwill or groundbreaking nature of the Pi. In that light, the twin lessons of managing customer expectations and ensuring the readiness of external suppliers have to be learnt and implemented. Without them, a product lacking the Pi's star qualities risks sinking without trace.

Note: Comments are moderated for this piece. Be civil if you do comment, disjointed fanboy rants will be derided. I'm a Raspberry Pi fanboy who's just as excited about the project as you are.

An AJAX and jQuery driven web feature, the OxfordWords Text Analyser

    If you are a follower of the OxfordWords Blog, you may have seen the launch of the OxfordWords Text Analyser, coinciding with the 200th anniversary of Charles Dickens. Here follows a technical description of the feature, what it does and how it works.
    The challenge was to show the logophile visitors to OxfordWords some of the computational linguistic techniques used in the preparation of the Oxford English Corpus, and thus give some insight into the preparation of a modern dictionary.
    Finding ourselves unable to share the corpus itself with the public, we settled on the idea of  delivering a much smaller text with similar analytical techniques applied to it to those used by our corpus analysis software. Since we already publish a huge range of classic texts in the Oxford World's Classics range it made the most sense to use those as our sources and provide collocate and frequency analysis as well as example sentences for each text. This presented a data problem: to incorporate all this in a single web page would mean adding several megabytes of data to the page, resulting in an unsustainable page load time.
    The obvious solution was to create a lightweight page containing just the Javascript display code, and deliver all the data on an as-needed basis via AJAX calls. There was a time when writing this to work reliably on all browsers would have been the bane of a programmer's life, but fortunately we now have the jQuery library to abstract such nasty jobs from the programmer, so taking that route was a no-brainer decision.
    AJAX having been decided upon, the next decision related to the server-side component. I wrote a piece about this last summer, about how experience of database driven back-ends for language analysis had led me to precomputing data as json files rather than querying a database. Disk space is fast and cheap, server processing power isn't. So my next task was to write a set of command-line PHP scripts that generated a tree of JSON files for each word in the source text, containing collocates, frequencies and example sentences.
    While these tasks are essentially simple ones, they are quite computationally intensive. In a typical Dickens novel for instance, there will be as many as 20000 unique words, and every one of those words needs to be searched for across the whole text and the frequency of its colloscates in all the locations it appears in computed. The whole process takes about six hours, and produces a roughly 20Mb tree of thousands of tiny JSON files which are then uploaded to the web server.
    So that's the surprisingly low-tech backend for the feature, how about the front end?
    The core functionality of jQuery makes coding an application like this very simple. The three main parts of the feature live in hidden DIVs which are shuffled using the jQuery show() and hide() functions. TheJSON data is pulled in using jQuery's getJSON() function. Collocates are shown in a word cloud courtesy of the excellent jQCloud plugin, example sentences are simply loaded into an unordered list, and frequency graphs are created using Google's Image Chart API.
    These plugins and code made a working feature. But to make a single-page jQuery application like this one feel like a proper application, there was one further component required. Users expect to be able to use the back button, and to be able to return to a particular part of the application by URL alone. Miss Havisham's dress needs to be readily conjoured up by linking directly to the word "bridal".
     We thus used the jQuery-BBQ plugin to provide URL and history functionality through the use of in-page anchors. Because the page can't be reloaded, the plugin appends the word to the end of the URL after a # symbol as it triggers any changes to what is displayed.
    In summary, the use of precomputed JSON files made the server-side compontnet of this feature very simple at the expense of using more filesystem space, and the use of jQuery and its plugins made client-side development a lot faster and more reliable across browsers. Sometimes libraries like jQuery are used for the sake of it when simple Javascript would have sufficed, but in this case I believe its use made for a far better application.

I call it stealing

    I have a friend who receives Disability Living Allowance. She can walk for about fifty yards, then she has to sit down, such are her medical problems. The DLA is paid to her so that she can keep herself mobile, it pays for the maintenance on her mobility scooter as well as the numerous taxi fares she needs to get herself around town.
    Her scooter is showing its age though. The gearbox is making some alarming noises and she thinks it's about time she replaced it. No problem, the DLA will catch that and the people who service her old scooter have a handy scheme whereby for a not-too-large monthly outlay she can lease a brand new one, all bills paid.
    She's no engineer, so when presented with a brochure containing reams of specifications she called me to help make some sense of them all so she could choose the right machine for her needs. No problem, so the other evening found me sitting on her sofa interpreting some of the sales patter.
    Along with the scooter paperwork was a catalogue containing all manner of aids for disabled and otherwise restricted people. Everything from specially designed drinking cups through bathing arrangements to electronic devices.
    As always I leafed through it out of interest.

    What I saw appalled me. Here were items I have seen for sale in 'normal' shops, yet with vastly inflated prices.

    A big-button mobile phone for nearly 200 quid, that I've seen on the High Street for under 50 quid. A mini CCTV system for more than I've paid for some cars, that I've seen in the Maplin catalogue for around the price of a cheap bicycle. Those are the two that stuck in my mind because I knew exactly the items in other places, but there were plenty more rather expensive looking but suspiciously simple gadgets.
    We've all seen cases where some groups end up paying more for items than they should. Computer peripheral manufacturers should be ashamed of themselves for adding a mark-up just because a PC product is targeted at mac users, for instance. But mac users usually have plenty of money and often have the capability to think for themselves that they're being ripped off.
    In this case an industry is specifically targeting vulnerable people who may not have the capability to shop around, and who are grateful for anything they may be able to lay their hands on. Such a huge markup for that customer base in immoral and unethical.

    They probably call it business. I don't, I call it stealing.

Android vs. iOS apps: it's a matter of status

So my mission has been to research a category of smartphone apps. I've been to the iPhone App Store, the Android Market, the Windows Phone Marketplace, and the BlackBerry App World.

And I've seen a lot of apps, and a lot of screenshots. A hell of a lot of screenshots.

iPhone screen shots normally have the status bar included. The time, the battery and the connectivity status. Or 'iPod' if they were taken on an iPod Touch. You can level many criticisms at Apple, but you can't fault their design.

Microsoft and BlackBerry screenshots don't have a status bar. Their designs don't need one.

But Android? Oh dear. Fine, Android has a status bar. But for some reason developers uploading screenshots think it's perfectly OK to tell the world they've got mail. Or voicemail. Or missed calls, Advanced Task Manager, or a hundred other icons in their status bars.

Here's the thing, I'm an Android Market customer. As such I have the attention span of a goldfish, I'm not paying you any money for your app and I owe developers no loyalty. Therefore if your app fails to grab my attention in the first couple of seconds after page load, it's history. So if the screenshot looks a little crap, the developer loses a download.

Amazingly it seems the majority of Android developers don't get this and fail to clear their status bars and turn off their connectivity before taking a screenshot. Just for fun I pasted a few status bars together in the image alongside this article.

I downloaded them because I had to to complete my task. But would you?

FAILBlog #fail

    My life is complete. A piece of software I wrote for the Oxford Words Blog, an online Shakespearean vocabulary checker, has been featured on FAILBlog.
    Pretty funny, the FAILBlog submitter has picked up on the fact that if you put the word 'balls' into the form repeated more than 20 times it tells you your English is 100% Shakespearean. Which though undeniably entertaining is entirely true, since 'balls' was a word used by the Bard.
    Unfortunately though it is not a fail, unless you didn't read the page and have misunderstood the purpose of the script. It is not there to deliver a verdict on the quality of writing, merely to correlate the vocabulary with that of the Bard. Which it has done, exactly as it should.
    I can't help feeling FAILBlog have failed though so, I posted the following comment:

First of all, thank you FAILBlog for featuring this. It has always been an ambition of mine to have a piece of my software featured on your site, and now I have achieved that goal.
I wrote the script featured in the screenshot above, you can find it at http://blog.oxforddictionaries.com/2011/08/how-shakespearean-are-you/.
At first sight it does look like a fail, but as it says on the page this is not a script for judging the quality of writing, it is simply a vocabulary correlation checker that compares the vocabulary entered with that of the Bard and delivers a percentage result. So since the Bard did use ‘balls’, it is exactly right in saying the text is 100% Shakespearean in vocabulary, because it is.
But that does not make the odd-looking result any less entertaining, so please feel free to have a laugh at our expense.

     I guess the guys at FAILBlog don't appreciate it when they get something wrong and the originator makes a comment about it, because their moderators have chosen not to publish my comment.
   
Edit - several hours after I posted this piece and hours after publishing a load of other comments the FAILBlog people published my comment. Well done! Did I stumble on an editorial approval policy?:)

How smart does a smartphone have to be?

    It started with an argument I heard last year at an event for mobile application developers. On my left, a developer, on my right a mobile phone industry maven. The developer was waving his three-year-old Nokia E71 and using it to illustrate a point about the smartphone market.
    The maven remarked that in the context of the smartphone market in 2010 the Nokia was not a smartphone even though it meets the basic criteria for being a smartphone of having a general purpose operating system on which the user could install their own software. His reasoning was that it was not marketed as a smartphone in the same way as iOS and Android devices have been and it lacks the readily accessible app stores available on those devices, and most users would not use an E71 in the same way that they would use an iPhone. They would use the E71 as a phone while they'd use the iPhone as a computer.
    A lengthy argument ensued with the developer demonstrating the apps on his Nokia and the maven simply pointing out that a developer prepared to source Symbian apps and install them himself is hardly the typical mobile phone user.
    I can so see how both sides of the argument have merit. Any Symbian phone is a smartphone in the classical sense, but the differing experiences between using the Nokia and using a modern phone running Android or iOS mean that most users of Series 60 phones like the E71 would have been just as well served by a series 40 feature phone, such was their low uptake of the smartphone features.
    I've been given cause to rememeber the argument a couple of times this year, as I've left the charger for my Motorola DEXT Android phone at my parents house and had to revert to using my previous phone, a Nokia N73. I'd never fully made use of the N73's smartphone facilities when it was my only phone because I didn't have a data plan worth anything, so how easy would it be to use a series 60 device as a smartphone in 2011? I installed the Motorola's SIM with its hefty bandwidth allowance and set the N73 to work.
    At the very start I knew there were areas in which a five year old phone could not compete. The N73 is one of the best voice phones I have ever owned and it has an excellent camera, but events have moved on since it was my fancy new toy. There is no touch screen so input is from a numeric keypad and thumbstick, no wifi or GPS, and its rather archaic miniSD card is limited to only 2Gb. It seemed so large when it was new! So I was interested in two areas only: what is the ease of finding and installing software on a Series 60 device and can the device and software do the jobs I demand of my smartphone?
    That posed the obvious question: what do I use my smartphone for? Like most Android or iPhone users I have installed a variety of apps for all sorts of purposes. I guess you could call me an Android power user, the DEXT has been rooted and sports a CyanogenMod community ROM.
    Some of the apps on the DEXT such as the compass app or the GPS-driven OS map app were not applicable to the N73 because of the lack of GPS. Others I might have installed but never really used much. So I thought about it and decided that I use the Android phone most for web browsing, Gmail and Google Maps. All tasks that should be within the capabilities of a device like the N73.
    As soon as I turned the N73 back on earlier this year I realised that there was another issue to consider. Not one that would prevent the phone being used but nevertheless a big consideration when comparing with a modern smartphone: Symbian is a capable modern OS under the hood, but Nokia's interface feels so much like one created for an appliance while the iOS and Android interfaces feel like those of desktop computers. You might excuse this by the age of the device, but having recently had a go with my wife's brand-new Nokia N8 I'd have to say that this is still the case with their most recent version. This is not necessarily a bad thing but does reflect the culture behind it, Nokia are a manufacturer of dedicated devices while Apple and Google come from a general purpose computing background. Another point for the maven above's premise that the marketing for Symbian devices never really capitalised on their smartphone abilities. Perhaps if they had, Nokia wouldn't be in the position they are now.
    So I had an old phone ready to go. Fortunately I'd charged the battery every month or two so everything still worked. My first step was to plug it in to Ovi Suite and run the software updater. At this point I hit another area in which the Nokia fails to compete with other smartphones. This was the very latest version of Ovi Suite as installed for my wife's N8, you'd think that by now Nokia would have employed a user experience designer to make it easier to use. By comparison with similar tasks on an iOS or Android device which Just Work, Ovi Suite is painful to use, a piece of annoying bloatware.
    New firmware in place, and what did it do for me? Nokia Maps was new, so it was back to Ovi Suite to download the UK maps. Yet more tedious fiddling. Then to look at the apps.
    Here I met the biggest let-down and the confirmation of the maven's view over that of the developer. Comparing the choice of apps on an Android phone with those in Ovi store reminded me of my Amiga days nearly twenty years ago when I would see poor quality software being punted for several times the price of its equivalent for the PC. How many idiots really pay five quid for a novelty theme for their Symbian phone? Yet that seemed to be most of what was on offer in the app store, to find anything useful I had to resort to Google to find individual developer sites.
    Fortunately though, I didn't need to look very far for my main apps. The updated firmware gave me a newer version of Nokia's Symbian browser but that hardly delivers a modern browsing experience. The same goes for the Nokia email client, in a cloud email world it just seems so dated. So my web browsing was always going to be taken care of by Opera Mini and email was well catered for by Google's native S60 Gmail app. I also took the time to install Google's search app and Google Maps app.
    So my viewport on the world became a 240x320 pixel portrait display and my input device a numeric keypad and thumb joystick. I needed to relearn the art of entering text using a keypad and resist the urge to use my finger on the display.
    My respect knows no bounds for the developers of Opera Mini. To fully replicate a modern desktop browser with server-side rendering on a very basic phone was always going to be a tall order, but they have come as close to that goal as they can, delivering a surprisingly useful web experience that comprehensively shames even the latest version of Nokia's own browser on the N8. I can check my work email using the web version of Outlook, I can easily browse full desktop versions of most of my favourite web sites and I find the interface squashed into the tiny screen of the N73 to feel modern, be intuitive and easy to use.
    The same can not quite be said of the Gmail client interface. It's easy enough to use but it has a very different feel to that of Opera Mini. It uses the Symbian GUI features heavily, and as such it feels like something running on a phone from 2001 with a grey LCD rather than one from 2006 with a colour TFT.
    Having spent a couple of weeks back in N73 country I have to admit I found it perfectly adequate for 2011. Sending a Tweet or replying to an email is a little tedious through a numeric keypad but using the joystick to control the pointer soon became second nature. And the Nokia's far superior hardware meant I had a signal in the most unlikely of places where the Moto would have been a relatively useless brick. I've posted pictures to Twitpic, made forum posts, referred to an email from my boss in a meeting and sent a drunken Tweet from a pub looking for a quiz team. In short, the N73 has been my constant companion in exactly the same way as the DEXT and though it can't quite do all the things the Android phone can, it's been useful enough for me not to become frustrated with it.
    On the whole after a while using Series 60 again I agree with the maven's view above. It is most definitely a smartphone OS and I've used it as such, but get it to do so I've had to search the web for apps and navigate a rather tedious and unintuitive PC software suite. But I can see the developer's point of view, it's been a rock solid small computer for some quite intensive mobile browsing and email on the go. Even with CyanogenMod I'd have had to reboot the DEXT a few times in a couple of weeks, by comparison I never had to reboot the N73 to make it perform properly. The only thing I found tedious with the Nokia was its text input, but then again I never got on with numeric keypads. If I get the chance I'd like to try a Bluetooth keyboard with it, or perhaps a QWERTY device like the E71.
    So I'll not be giving up the Motorola. It is a better computer, even if not always the better phone. But the Nokia has been rescued from the drawer and kept charged, because I now realise it's too useful to ignore.

Precomputing for fun and profit

    Last year I wrote a post about my keyword analysis tool. I wrote this piece of software in the years before I started working for OUP, and only now I have seen the tools at the disposal of the lexicographers do I realise that I created what was in effect a corpus analysis tool.
    I used the technologies I was familiar with using for web sites, PHP and MySQL. These are the bread-and-butter of web development, making it possible to create dynamic and flexible web content engines with relative ease. Since an SQL engine is designed for the querying and analysis of data, I thought, MySQL would be perfect for the task of language analysis.
    I was proved right, to a point. I was able to quickly hack together a fully functional tool that would analyse a corpus to provide me with collocates and their frequencies for any n-word phrase. But in sticking dogmatically to what I knew, beyond that point I had taken entirely the wrong approach. I was using a very powerful database engine to perform the same very complex task again and again on an extremely large data set, and as my corpus edged into the tens of millions of words I found that my analysis was redolent of the early days of mainframe computing when programmers would start a job and go away to have a cup of tea while it ran. The system became so slow as to be unusable. It works, but even with all the MySQL optimisations in the book it does not do so in the real time demanded by today's users.
    My mistake was to rely on an expensive commodity when I should have used a cheap one. When I first used computers twenty-plus years ago, storage was the expensive commodity. We used to have disk compression systems using a cheap commodity - the processor time on our PC-ATs - to expand an expensive commodity - the free space on out 10 and 20Mb hard drives. Thus we were trained as programmers to be as efficient as possible in our use of storage space and memory.
    Space efficiency is a good thing if you are programming for a tiny embedded system but in 2011 disk space on web servers is laughably cheap. Processing power, as represented by the waiting time of your users, is not. Clearly a rethink is called for.
    So as an experiment I tried precomputing collocates and frequencies a subset of phrases and saving them as a structure of separate tiny JSON files. A simple JQuery based browser interface was then cobbled together to browse them, resulting in an application that was near-instantaneous for analysis that would have taken minutes to compute all the separate component queries using the MySQL tool.
    The moral of this story is I guess to stand back from your choice of underlying software tools and ask yourself whether they have been driven by suitability or familiarity. I used PHP/MySQL because I am used to it, I never gave a thought to whether a simpler solution might not be much better.
    So I'll be revisiting my hacked-together precomputing engine and applying it to my entire corpus. I may end up with a few gigabytes of JSON files instead of a large MySQL database but to the user that doesn't matter.
    I never thought I'd be returning to flat file storage, in fact I've mocked people who use perl DB files for similar tasks in the past. Guess I'd better eat those words then, precomputed of course!

The annoyance of writing stupid things

   It is with some surprise that I notice six months have passed since I last published anything here. It's not that I haven't written anything, I see I have four draft posts stored up, or that I haven't been writing things elsewhere, just that what I have written here has seemed so banal when reviewed that I haven't pressed that 'Publish' button.

   Better to say nothing than say too much and confirm to the world that you are a fool.