townx

I have a Nokia 770. It's a nice Linux-based PDA, with a Debian core plus some proprietary bits and bobs (e.g. Flash, RealPlayer) incorporated. I use it at home if I leave my laptop at work for simple web browsing and checking email. The main bonus being I can use it lying down :).

A couple of days ago, the latest firmware was released. As there were major improvements (e.g. fingertip input support, password saving in the web browser, GoogleTalk integration), I thought it was worthwhile to upgrade. And there my troubles began.

At the weekend, I had installed a Windows 2000 VMware image, so at least I had the tools available without having to take home a different laptop. But as soon as I tried to install the Nokia 770 firmware update wizard, I encountered a hulking, rusty chain of dependencies:

• I wanted to use the Nokia 770 firmware update tool, which needs...
• .NET Framework 2.0, which needs...
• Microsoft Windows Installer 3.0, which requests...
• Your Windows serial number to enable Windows Genuine Advantage; which needs...
• Service Pack 3, which necessitated...
• Windows Update, which required...
• A new version of the Windows Update software to be installed

On my home machine, it took about an hour (over slow ADSL) to download all the bits of software, find my CD case with serial number, and install everything. The Nokia update took another 20 minutes on top of that. Hmmm, don't think I'll be switching to Windows any time soon. Granted, I am running an ancient version of Windows, but so are lots of businesses who, like me, can't afford to upgrade. It's so much easier on Ubuntu, with a nifty apt-get install. Anyhow, I suppose it's worth the effort to upgrade the Nokia, as the improvements make a big difference to usability.

Update: to add insult to injury, when I tried to follow the same procedure to update Jono's 770, it completely screwed the firmware, so it won't even boot. What is wrong with treating the 770 as a big USB hard drive and dropping the new firmware into a specially-named folder? (This is how I upgrade the firmware on my iAUDIO M3 - nice piece of kit, by the way, except I think the headphone jack socket is wearing out through over-use, and I get occasional wierd freezes which require a reset.)

A while ago I write a thing called WAML, a small library for writing an HTTP client in Ruby, with support for tidying HTML, simple XML parsing, creating RSS feeds, and sending files over SSH (I wrote it to screen scrape pages, push them into RSS files, then send the output files to a website).

Like a good Rubyist, I included a whole bank of tests, including a few for the HTTP client stuff. However, at the time, I just pointed at pages on the internet, and my HTTP client tests would dutifully march out into the wilds to pick up web pages. This is not ideal: what if a website is down? or it changes? or I don't have a network connection?

I found a really neat solution called FakeWeb, which solved all my problems. This interposes between your testing code and the Net::HTTP libraries (the main HTTP libraries for Ruby). So you can run your test code against a "fake web": any HTTP requests sent through Net::HTTP are intercepted by FakeWeb and handled by it instead. Any pages you want to "mock up" are registered with FakeWeb using the register_uri method, and you can attach arbitrary headers, cookies, and page content to each registered URI. Here's an example of how I applied it, starting with my original test, which checks whether my HTTP client correctly sets cookies on redirected requests:

class TestHTTPClient < Test::Unit::TestCase
  def setup
    @client = Client.new :user_agent => 'Firefox'
  end

  # test passing of cookies gathered from one response to subsequent request
  def test_cookie_passing
    response = @client.post(:absolute_url => 'http://bloglines.com/login', 
    :post_parameters => { 'r' => '/myblogs', 'email' => 'elliot at townx.org', 
    'password' => 'notmyrealpassword', 'Sign In' => 'Log In'})
    
    assert_equal 'http://bloglines.com/myblogs', response.url
    assert_equal 'elliot@townx.org', response.cookies['BloglinesEmail']
    assert response.body.include?('<frame src="/info/myfeeds" name="basefrm" scrolling="yes" />')
  end
end

(I've left out all the requires to keep it neat.)

Note that this code physically logs into BlogLines by sending a post request, then checks that the client correctly follows the redirect to the /myblogs page and has a cookie set. There are a couple of problems with this:

• You need a network to run the test.
• It exposes my username and password to anyone I distribute the code to.
• It hits Bloglines unnecessarily.

Here is the test, rewritten using FakeWeb, which fixes these issues. More verbose, yes, but friendly. Plus I can test any kind of arbitrary series of web pages without having to setup a special website to do so:

class TestHTTPClient < Test::Unit::TestCase
  def setup
    @client = Client.new :user_agent => 'Firefox'
    login_setup
  end

  # setup fakeweb to emulate login procedure
  def login_setup 
    base_url = 'http://pretend.web.com/'

    # URL where the agent will try to login
    @login_url = base_url + 'login'

    # URL to redirect to after login page
    @bloglist_url = base_url + 'myblogs'
    @bloglist_content = '<title>Logged In</title>'

    # email address to use during login and in cookie
    @email = 'elliot at townx.org'
    
    # custom build response from the login page, to redirect to the bloglist and set a cookie
    redirect_response = Net::HTTP::HTTPResponse.new('1.1', '302', 'Found')
    redirect_response['Location'] = @bloglist_url
    redirect_response['Set-Cookie'] = 'email=' + @email + 
    '; expires=Sun, 31-Jul-07 15:22:12 GMT; path=/'
    
    # register the two URIs with FakeWeb and attach their content
    FakeWeb.register_uri(@login_url, :response => redirect_response)
    FakeWeb.register_uri(@bloglist_url, :string => @bloglist_content)
  end

  # test passing of cookies gathered from one response to subsequent request
  def test_cookie_passing
    response = @client.post(:absolute_url => @login_url, 
    :post_parameters => { 'r' => '/blog_list', :username => @email, 
    'Sign In' => 'Log In'})
    
    assert_equal @bloglist_url, response.url
    assert_equal @email, response.cookies['email']
    assert response.body.include?(@bloglist_content)
  end
end

Note the quick and dirty way I setup the response returned by the login page, to emulate a redirect with a cookie (I deserve a slapped wrist for that baby, especially the manual date setting). I will probably extend FakeWeb to do this better. Plus I had to add a new method to Net::HTTPResponse to allow me to manually set a response body. But it works pretty nice now. It would also make sense for me to rewrite my FlickrLilli tests like this (they currently use Flickr directly, which means the tests can take a while).

The WAML gem is attached to the bottom of this post, for anyone interested. I haven't updated it for a while, but it still does its job.

Can anyone recommend me an HTTP client library for Ruby? I know about Net::HTTP, but that's pretty low level: I'm looking for something like Apache Commons HTTPClient. I've written my own, but it's primitive. I basically need stuff like:

• Following redirections automatically (to some pre-specified depth)
• Automatically attaching cookies to requests (my current code does this, but not on a per-host basis, and not respecting cookie expirations, paths, secure etc.)
• Enable me to parse HTML, tidy it, and extract parts of it

If there's nothing out there, I'll carry on building my own (codenamed: WAML [Web Automation Macro Language]). But it would be interesting to find out if anyone else is working on this already before I go too far.

I'm still trying to get my thoughts in order for a forthcoming seminar on making money because of open source. I found this short article on RFID, which encapsulates the arguments for standardisation and its effects on the market really concisely:

...standards won't turn RFID into a commodity. Standardization will increase competition, drive innovation and lead to products with meaningful differences.

We've seen the same thing happen with the web server market, and currently happening with content management. Not that there are meaningful standards among content management systems (at least, not PHP ones), but the fact that most CMSs provide equivalent base functionality has meant differentiation has had to move to other areas of the market: e.g. better user interfaces and better integration with external systems.

I enjoyed OSBC last week, though I thought it was a bit pricey for what we got. Simon Phipps' presentation was very good indeed (plus he attacked IP indemnification firms as pointless scare-mongers); I enjoyed Trolltech's Eirik Chambe-Eng laying into Symbian, stating that Linux can scale down better and is more attractive for low-end phones (some bloke from Symbian stood up at the end, a bit miffed); Ian Howells from Alfresco had some interesting points about how the Enterprise Content Management market is ripe for exploitation by open source (monolithic, 15 year old code bases in existing systems are the opposite of agile); Mark Fleury bristled through the panel sessions; and Brian Behlendorf said some interesting stuff about using open source style development in large software houses (though the audience wasn't particularly responsive). Also met a couple of nice chaps from Sun.

A few observations:

• The presence of masses of people from Symbian gives a hint that they are a bit scared about what open source might do to the phone market.
• The audience seemed polarised between people and companies who "get" open source (Brian Behlendorf, Trolltech, Alfresco, Funambol, JBoss), and dived in long ago; and companies who are dipping their toes and either running away because the water's cold or standing there expecting armbands to be provided by "the community" (Microsoft, obviously; J.P. Morgan; Philips Medical Systems).
• It's a bit distressing to see that IP issues are still the main stumbling block for larger firms, and that talks with titles like "Risks Propriety Software Vendors Face in Using Open Source" still attract a lot of people. I feel like the issue is not really about protecting yourself from open source: it's more about joining the party with the right attitude. This is where Sun have got it right by employing a "conscience" like Simon Phipps.
• I attended one session about open standards, where proprietary vendors were attempting to defend standards with royalties attached. Why can't we have royalty-free standards (good for consumers), and let proprietary vendors compete with proprietary standards if they choose to? Would the internet be the massive monument to human ingenuity it now is if it had been proprietary? (gopher, anyone?). I would love to see the proprietary, non-free "standards"-based GSM mobile phone market (for example) decimated by cheap phones based on Linux using VOIP over WiMax. (I hate mobile phones in their current incarnation: the antithesis of free.)

That's all I have to say on the matter. And no, I will neither forgive nor forget Hazard.

Update: This has mostly been superseded by my full-on s33r project, hosted on RubyForge. It contains a Rails application as an example, but far more functionality than this early effort. I'd go so far as to say it outstrips the Amazon Ruby sample code for S3, as it provides object wrappers around the bucket listing, logging and acl functionality, making it much easier to utilise them than it is with the Amazon samples. Give it a try, why don't you?

I am in old skool Detroit techno heaven: I found out about Detroit Digital Vinyl yesterday, and downloaded a few tracks last night by the mighty Underground Resistance (Death Star - what a classic of brooding, disorientating acid that is), plus the compilation Interstellar Fugitives. It worked out at £7.50 for the lot, which is about right for me, given that this stuff is pretty hard to get your hands on otherwise. I've been hankering after this stuff since reading some time in the '90s that the Aphex Twin is a big fan; plus I had a lot of old acid tracks (DJ Pierre, Phuture, Tyree, Mr. Fingers, Adonis) on crumpled tapes and needed a fresh acid/house injection. Recommended.

A few things have prompted me to think about how open source is gradually being recognised as "enterprise-ready":

• Articles in IT Week and Computer Weekly give a nice summary of the OSDL's survey of why companies deploy Linux desktops. Surprisingly, the respondent companies' top reasons for going with Linux on the desktop were due to pressure from employees, and due to successful roll-outs by competitors. In one of my talks, intended for IT SMEs, I use one of these arguments to try to persuade SMEs to go with open source: "Look, here are some successful companies like you who are already using open source; why aren't you?". It seems odd that this is a better reason for doing something than for other business reasons, like cost, stability, security, removal of vendor lock-in, etc.. But there you go: it seems to work for the companies I deal with, as well as enterprise types looking to do Linux roll-outs.

• Computer Weekly runs a series each week called "Hot Skills", where they highlight skills IT professionals should learn to make themselves employable. What's interesting is that for the past half dozen episodes, the technologies have been open source: PHP, Python, Linux, LAMPP, PostgreSQL. Contrast this with the bulk of the paper, where open source is frequently completely absent, and there is a gross emphasis on monolithic dinosaur companies and their god-awful, horrifically expensive, locked-in, old-fashioned software.

• My article "Why you should care about open source" is now available online as part of the British Computer Society's Annual Review 2006. Looking at the rest of the articles, there are a good number about open source; compare this with last year's review, where none of the articles had "open source" in the title.

So open source is (slowly) gaining more acceptance in the mainstream (enterprise). My feeling is that it's not so much acceptance as recognition: I reckon most decent/big companies use open source somewhere, whether they know it or not, and this wave of interest is simply highlighting the fact that executives have realised this and are formalising casual use. Open source was there all along: big companies are only just now admitting it.

This is how to install the APC caching extension (I got version 3.0.6) into XAMPP on Linux (for PHP5). You will need the developer package for XAMPP to do this. (You can use the pear command to install it if you're using standard Apache + PHP.)

• Download the tarball from http://pecl.php.net/package/APC
• Unpack it and connect into the directory created.
• Run /opt/lampp/bin/phpize-x.x.x; use the version of phpize which corresponds to the version of PHP you want to install for.
• Run the following commands:
  
  ./configure --enable-apc --with-apxs --with-php-config=/opt/lampp/bin/php-config-5.0.4

make
make install
• Edit your php.ini file and add the following line
  
  extension=apc.so
• Reload Apache.
• Create a PHP page which calls phpinfo(), and check that there is a section for APC. If there is, it's working.