Deadnode

New Platform

2010-03-10T00:00:00+00:00

With the dawn of 2010, I decided it was time to move away from Radiant as the CMS for most of the sites I host and maintain. Radiant does a reasonable job, with a nice simple user interface non-technical users can grasp quite easily, but it takes a great deal of memory and slows down the site being served with all the overhead of going from the web server, through Mongrel or FastCGI into Radiant then rummaging around in MySQL to retrieve all the page components.

Radiant’s caching helped – and, in conjunction with Lighttpd’s mod_magnet facility, I was able to add a simple Lua script which would turn any recent cache hit into a simple static content delivery through Lighttpd, bypassing Radiant entirely – but for sites with very infrequent changes, having over a hundred megabytes (on a server with 360Mb total!) tied up in a CMS whose features weren’t used from one week to the next seemed silly.

In particular, after an unexpected reboot of the server which holds this site and a few others, the two Radiant instances each took over 100 Mb of RAM – which, on a server with 360 Mb total and several other services active, meant hitting swap and several minutes of disk thrashing before the sites were live again.

At the time, I had one Radiant instance hosting the SUE-MoT project website, with a second hosting the remaining sites including this one. With the SUE-MoT project having officially ended some months earlier, there would be no significant updates – perhaps no updates at all – this site was an easy one to migrate: a simple wget command was enough, with a second to retrieve the current sitemap XML file.

For the other sites, more actively maintained, it’s a little more difficult. I decided to migrate this site second; rather than taking a static snapshot as for SUE-MoT, I wanted something that would keep updates nice and simple. A few options came to mind: rebuilding the pages using something like SSI to pull in the common elements, or running the files through a pre-processor of some sort to generate static content to upload.

Enter Jekyll: a simple Ruby tool for generating static content from templates and text files, with Textile support, so I could simply copy most of my content directly from Radiant into Jekyll files, layouts to simplify editing common elements the same way I could with Radiant – all that’s missing is the dynamic content (recursion etc) with Radiant’s inline tags starting <r:.

It isn’t quite perfect – very limited support for generating content such as page lists, and documentation which leaves almost everything to be desired – but it will do the job I need for now. In the longer term, I’ll find or develop a replacement, probably not using Ruby and hopefully having at least basic documentation for all the features. The SUE-MoT site relied on simple Radiant iteration and conditional statements; the Jekyll documentation suggests such things exist but fails to give enough information to use them, which forced me to use wget instead. Fortunately, this site never used those – just a hard-wired list of headings in the page template – and there are clever ways to achieve similar results quite elegantly with CSS selectors which I will use and document in the near future.

Windows woes: STOP 0x0000007b after changing partition sizes

2009-11-28T00:00:00+00:00

Last week, I found myself short of space on my Windows Vista partition on my main development PC. Originally, that machine dual-booted Windows XP x64 and Windows Vista x64, each from their own partition on one fast (10kRPM ‘Raptor’) SATA disk, with data living on two other disks. This meant a simple procedure: shrink the old XP partition to a few hundred Mb (I couldn’t delete it entirely, since the Vista bootloader lived there) and enlarge the other partition to fill the new space. Simple enough using any of the partitioning tools; I chose Paragon Partition Manager in this instance.

All fine – until I came to reboot, and was confronted with the dreaded STOP 7B error (‘inaccessible boot device’). Windows was able to load the kernel and associated device drivers, using the simple boot-time drivers, but failing when it came time to mount the file system properly and hook in the filter drivers. Contacting Paragon support and posting in their user forum got me a suggestion to boot from their recovery CD (which fails to boot on my system anyway), then delete both partitions and hope the partition undelete feature would recreate them in working form.

Google Voice: AT&T's real fear?

2009-08-21T00:00:00+01:00

Much has already been written about the rejection of Google’s official iPhone application from the Apple App Store. For some reason, Apple – at AT&T’s request, people suspect – refused to allow the application to be offered through the offical App Store, and indeed deleted unofficial applications offering similar facilities which had already been approved previously.

Apple themselves have no apparent motive for rejecting the Google Voice application; even if it used an undocumented interface, which is normally enough to trigger an app’s rejection, Google have done this before (in Google Mobile, with the proximity sensor) and Apple turned a blind eye. It isn’t Apple’s revenue stream Google Voice threatens, give or take whatever cut Apple get under the revenue sharing deal. Apple, on some official level, will have been aware of the unofficial Google Voice client applications when those were submitted, but still approved them, suggesting the functionality did not break any of Apple’s own rules, stated or otherwise.

AT&T, on the other hand, are in a very different situation. It has already been established that they influence the application approval process, which is why the Slingbox client was restricted to prohibit use over AT&T’s network, streaming only over WiFi. In that case, the change AT&T wanted was obvious: they prevented this use of their network, since without installing the Slingbox client you couldn’t stream that video. With Google Voice, the situation is very different: what would AT&T gain by blocking the installed native application, when you can use Google Voice through the web interface anyway? Unlike Apple, AT&T don’t see every single application submitted, only those which attract attention – for example, by coming from a multi-billion dollar corporation just breaking into the voice communication business. Once they started investigating, they would then discover the third party clients doing the same job, and want to put a stop to those as well.

Most analysis seems to assume it’s all about the calls, those precious minutes of airtime charges. Many seem to have assumed – wrongly – that Google Voice is a VoIP service, that Google Voice routes your calls over the data connection just as the (approved) Skype application does over WiFi connections. If that were true, AT&T would certainly have a reason to object – but it isn’t: Google Voice services are regular voice calls. Initiate a Google Voice call via the client application – native or web – and Google Voice triggers a three way call between itself, your handset and your destination, so AT&T get to charge for the minutes of airtime you’re using. It’s an incoming call rather than an outgoing one, but that has no impact on the charges: a minute of airtime is a minute or airtime, regardless of direction! Equally, blocking the application makes little difference: you can still use this feature through the web interface anyway.

Where the difference does matter, though, is text messaging. Sending and receiving text messages via Google Voice does indeed divert traffic onto the data connection, bypassing AT&T charges of up to $20/month – which, of course, are almost pure profit for them. You can still send messages through the web interface and escape these charges – but to receive any replies that way, you would have to have the web page open! Native applications can receive push notifications, which would allow your Google Voice text messages to pop up in exactly the same way as regular messages sent through – and charged by – AT&T. The web interface can’t do that: by blocking the native application, AT&T ensures that only their own expensive text messages have this important feature. Hit ‘reply’ to such a message, and that reply gets charged for by AT&T as well.

The only weakness of this hypothesis is that AT&T have not blocked the Google Voice client for BlackBerry. That is easily explained: where iPhones (unless jailbroken) can only run applications downloaded from the offical Apple store, requiring explicit prior approval from Apple to run – permission AT&T can persuade Apple to withhold or revoke – the BlackBerry simply downloads and installs applications over the Internet. Using RIM’s push notification requires going through their servers and paying them a fee – perhaps AT&T could push RIM to withhold that, but unlike the iPhone case, all this would achieve is preventing them from using push notification, not block the whole application; it also seems likely AT&T have less influence over RIM than they do with Apple in this respect, thanks to the big-ticket exclusivity contract.

Unsubscribing from McAfee/Network Associates AVERT

2009-06-04T00:00:00+01:00

McAfee offers quite a useful virus alert service by email – more details here – but unfortunately their mailing list is rather difficult to unsubscribe from. There are no instructions in the body of their emails at all; the headers do contain this promising looking entry:

List-Unsubscribe: <mailto:leave-512213-327810Xlistserv.mcafee.com>@

Frustratingly, however, although listserv.mcafee.com actually resolves and accepts SMTP connections, it rejects all delivery attempts for that domain:

RCPT TO:<leave-512213-327810Xlistserv.mcafee.com>@

550 Relaying denied.

The solution, it turns out, is to change the domain to listserv.nai.com – presumably the domain they were using when they first configured the mailing list software, and the only address it will actually accept. That domain resolves to the same server, 205.227.136.228, but this corrected address will actually be accepted. (McAfee was founded in 1987, then merged with Network General in 1997 to become Network Associates, reverting to the name McAfee in 2004.)

(The numbers are presumably subscriber-specific, so using this exact address will attempt to unsubscribe an old address of mine which has already been unsubscribed. Obviously, you will need to change these numbers to match those in your own email headers!)

Quite why a major company like McAfee is operating a mailing list with no functioning unsubscribe instructions, I don’t know, but hopefully anyone searching for help will now be able to find it, rather than having to guess the correct address!

If you don't built it, they may come anyway

2009-02-04T00:00:00+00:00

A perennial problem in maintaining widely-used platforms is backwards compatibility. Windows suffers more than most in this respect, both because of Microsoft’s unusual degree of commitment to backwards compatibility and the degree to which the platform has changed over time, from a 16 bit collection of libraries with cooperative multitasking running under MS DOS on a single 8086 processor with 640k to 32 or 64 bit pre-emptive multitasking operation on multi-core Xeons with gigabytes of RAM.

They’ve had to jump through endless hoops (NTVDM.EXE, anyone?) just to get those old applications running on operating systems created two decades later on radically different hardware. Further hoops, though, are imposed by applications relying on undocumented characteristics of components – sometimes, as Raymond Chen likes to point out, through laziness when someone just can’t be bothered putting in the effort to use the proper, documented method for something – but in other prominent cases like the MSI SlowInfoCache, an undocumented 552 byte binary block which controls how entries appear in Add/Remove Programs, and showing or hiding the Quick Launch bar, because Microsoft chose not to document an official way to do something out of fear “bad” people would use it. Needless to say, this just means they use undocumented methods instead!

Like many battles, there is merit on both sides. Within Microsoft’s walls, logic like Whether the Quick Launch bar is shown or hidden is an end user setting, and programs should not be overriding the user’s preferences. Explorer consciously does not expose an interface for showing and hiding taskbar bands because it would just be a target for abuse has prevailed. Because someone in Microsoft thinks programs (other programs, that is: clearly their own – Explorer, in this case – get treated differently) should not be tampering with this “user” setting, they choose not to make it easy for programs to do that.

On the other side of the fence, however, there are people who need, or at least want, to change that setting programmatically for some reason. Perhaps to impose a standard corporate desktop (many corporate IT administrators seem to believe “user” settings should actually be “Administrator” settings wherever possible, imposed centrally by them rather than allow mere end users any say in their work environments), or to implement some other feature – perhaps even one the users themselves have requested, having neglected to read Microsoft’s memo telling them they didn’t really want that.

The end result, of course, is usually that the feature gets implemented anyway. In the example above, you can use FindWindow to get the handle of Shell_TrayWnd, then send it message WMTRAY_TOGGLEQL (WM_USER+237) with WPARAM set to 0 and LPARAM set to 1 or 0 to enable or disable the Quick Launch bar respectively. Now, in this particular case, sending WMTRAY_TOGGLEQL messages seems unlikely to pose a future compatibility headache for Raymond and co to work around – but there are less straightforward examples out there: the MSI packaging system, for example, uses undocumented blocks of data 552 bytes long in the Registry (values named SlowInfoCache, under Microsoft\Windows\CurrentVersion\AppManagement\ARPCache) to interact with the Add/Remove Programs Control Panel applet. Since this block controls some of the details shown for your application’s entry in ARP, it is hardly surprising that people want to parse and manipulate this data – but being undocumented, how can we know there isn’t some undocumented flag in there which matters? With people relying on direct access to an undocumented binary block, extended that data structure in future will be painful as well.

Undocumented blocks of binary data. Just say no.

Caching with Radiant CMS, lighttpd and Lua

2009-02-02T00:00:00+00:00

Lighttpd is now one of the top five web servers: very lightweight and efficient, with FastCGI support – which is how my half-dozen Radiant websites are served – and a clever built-in Lua interpreter, allowing you to script modifications to requests in ways beyond Apache’s mod_rewrite. For each request, Radiant can cache the complete rendered page in a pair of files (one for the body, another for metadata) – with a short Lua script, you can then arrange for Lighttpd to serve up recently generated cache files directly without ever touching the Radiant process – an enormous performance gain.



if (string.sub(lighty.env[“request.uri”],-1)==“/”) then

        lighty.env[“request.uri”]=string.sub(lighty.env[“request.uri”],1,-2)

end

local file = “/radiant/cache/” .. lighty.env[“uri.authority”] .. lighty.env[“request.uri”] .. “.data”

local attr = lighty.stat(file)

if attr and (os.time()-attr[“st_mtime”])<3600 then

        lighty.header[“Cache-Control”] = “max-age=3600, public”

        lighty.header[“Expires”] = os.date (“!%a, %d %b %Y %H:%M:%S GMT”,os.time()+3600)

        lighty.header[“Content-Type”]=getmime(lighty.env[“request.uri”])
— Now load any header lines defined in the yml file:
meta=io.open(“/radiant/cache/” .. lighty.env[“uri.authority”] .. lighty.env[“request.uri”] .. “.yml”)
if (meta ~= nil) then
for line in meta:lines() do
— Extract header fields:
string.gsub(line,“^%s+([A-Z][A-Za-z-]):%s(.*)$”,function(key,val)
lighty.header[key]=val
end)
end
io.close(meta)
end
lighty.content = { { filename = file } }
print("HIT for: "..file)
return 200
else

        print("MISS for: "..file)

end

The 3600 on line 6 is the TTL: if the cached copy is less than 3,600 seconds (one hour) old, it will be served rather than re-rendering the page. If you have a single Radiant installation where all content updates are made through that server’s web interface, you can use a much higher value (obsolete cached files get purged immediately by Radiant when you edit that page) – Radiant doesn’t know to purge files which are indirectly edited, however (for example by editing a snippet), and if you replicate on the database level as I do, with a MySQL replication ring, the replicated Radiant instances will not be aware of the update, so they never purge those files. Lines 7 and 8 insert HTTP headers which convey a similar message to clients and proxy servers, indicating that this file is not expected to change within the next hour. If you want a more aggressive caching strategy, I would increase the first number (serving older cached copies from the server itself, if present, relying on Radiant purging the cache as needed) but leave the other two. The later part of this Lua script loads any other headers which Radiant has defined in the .yml file, most importantly Content-Type (in my early days using Radiant, I was bitten by this: serving up CSS files as text/html will make some, but not all, browsers ignore the CSS content completely). The regular expression deliberately limits headers to those beginning with a capital letter in order to disregard the “cookie: []” entry from the yml files.

Even the relatively conservative settings I have here make a significant difference – in particular, if a single popular page gets “Slashdotted”, almost every single hit will be served as a single item of static content: Radiant itself will only see one hit per hour for any given page, however popular.

I intend to try increasing the TTL significantly in the near future, having switched to a non-replicated configuration (more specifically, the content is still replicated on the database level between two nodes, but only one is actively serving content at any time) – I also have grander long term plans which should give me the best of both worlds, ensuring that obsolete content gets purged on replicas as well. In the shorter term, I would like to implement handling of If-Modified-Since requests: if the timestamp for Last-Modified in the cached object is older than the timestamp in the If-Modified-Since field of the request, I could return a 304 Not Modified status code and skip serving the content entirely. Compared to the performance difference between rendering a page through Radiant and serving a cached page as static content, however, this will be a trivial extra gain.

CloudFront - one month on

2009-01-19T00:00:00+00:00

After the first month of hosting most of my static content on CloudFront, I’ve been very impressed with the performance (which is, after all, the main reason for using a CDN).

There are a few minor irritations, or features which would be nice and seem obvious to me:

rsync access

Much more efficient for uploading from a local working copy. S3Fox’s ‘sync’ feature largely fills this gap; there’s probably a scriptable equivalent available somewhere.

Flat namespace

There is no true directory or folder hierarchy, although most of the front-ends provide some simulation of this. In particular, this could become an issue with large numbers of files: on a regular file server, you might structure a million files in a two-level directory structure, with a hundred directories and a hundred sub-directories each containing another hundred files – in S3, you’d really just have a million objects in a single ‘bucket’ (S3’s top-level container). This might not bother S3 itself, but could be problematic for other tools. It also complicates permissions a little: rather than granting permissions to a single parent, you need to modify the access control list of every single file. Granting or revoking access to that million file collection would probably entail two million metadata operations (one to fetch each ACL, another to write the new one back) – racking up something like $20 in AWS charges and probably taking many hours!

Content listings

Permissions are a little different to the Unix hosting most of us will be familiar with. After uploading my first batch of content, I told S3Fox to grant anonymous read access recursively to my CloudFront bucket as a quick way to make the objects I had just uploaded accessible. On a standard Unix system, you need X (execute) permission on a directory to access its contents; on S3, all you need is read access to the object (file) itself. Read access to the bucket actually enables access to an XML list of the bucket’s contents, which you probably don’t want.

No DirectoryIndex

If you want a file like index.html served, you need to link to it explicitly: no linking to hostname/foo/ if you really want hostname/foo/index.html served, as you would expect from a standard web server. Having said that, uploading your HTML file with an object name of “foo/” may be sufficient: there are no directories in S3, but slashes are valid parts of object names instead. I’ll experiment further on this later and report back.

Expiry and versioning

Everything has a minimum TTL of 24 hours when fetched by CloudFront, which you can increase by setting appropriate headers on the object in S3. Most CDNs set long TTLs for efficiency’s sake, although those like CacheFly employing push-based replication may not need to.

Unfortunately, the current version of S3Fox doesn’t give access to custom headers, but S3Hub is a good S3 client for Mac OS X which does – I have now set 2019 Expires headers and a 365 day TTL for caches on all the versioned objects on cdn.deadnode.org.

The trick here is to ensure that whenever an object is changed, the URL by which it is referenced also changes. Either embed a hash of the object in the URL (MD5 or even CRC will do: it’s to guard against near-contemporary versions of an object having the same URL, not to provide actual “security”), a monotonic counter or a timestamp.

I’m currently using the first two approaches: the CSS file for this site is presently style-v4.css and will be probably replaced by style-v5.css at some point in the future; the banner image at the top is /1318a51d1a358a1aa49ccabaa7f48fad/header.jpeg where the string of hex digits at the start is an MD5 hash of the contents.

Cost control

One noticeable omission is any kind of cost control. Most hosting packages I’ve looked at recently include some threshold at which they will warn you before you rack up outrageous charges; AWS seems to ensure that your site getting deluged with a sudden influx of hits won’t overload the servers – the notorious “Slashdot effect” which reduced so many popular sites to a collection of error messages in years past when a site suddenly became popular – but leaves you open to a wallet overload instead. Having said that, the prices are low enough that even a relatively large video file getting thousands of unexpected visitors should only be a matter of tens rather than hundreds or thousands of dollars.

Dollars

The price above was in (US) dollars for two reasons: first, it’s a currency more visitors to this site are likely to understand than any other – but secondly, it’s currently the only currency Amazon charges in. You need to be aware of this if, like me, your cards are in any other currency, both because this means Amazon’s prices effectively fluctuate randomly to some extent, and because I have occasionally been burned: one of my Visa cards applies a significant surcharge to all transactions in a foreign currency.

No Edge-Side Includes

It’s apparently static content only, while some CDNs – most notably Akamai – offer the assembly of content at the edge. Using ESI – an open W3 standard to which Akamai, IBM and others contributed – you can build a dynamic page on their CDN servers, with all the static portions of the page (even the most dynamic page will still contain a large amount of static data) being cached in the CDN rather than fetched from your own origin servers.

Referrer restrictions (hot-linking protection)

There also seems to be no way to protect content from hot linking racking up large bandwidth bills. Most of the cheaper CDN options seem to lack this, presumably because of the extra overhead involved in setting it up, but this would be a valuable feature; at one point some years ago, I discovered almost half my total bandwidth bill resulted from one person having linked directly to an animation clip I hosted as her avatar on a popular online forum – so every single viewer of any page she had posted on was downloading that fairly large clip from my server. At the time, fixing this was a one-line change to my web server configuration file; on S3/CloudFront, this could have been an expensive ongoing problem.

Experience so far

The hit rates so far seem quite impressive: from the S3 billing figures, well over 90% of my traffic is served up from the CloudFront nodes’ caches rather than being fetched from S3. With clusters on three continents (one in each of Hong Kong and Japan, eight in the US and four in Europe) access should be very fast for almost everyone, although Australia still seems an unfortunate omission for now. It is very good value, particularly at the low end – a nice clear pricing structure with no commitment, but the ability to scale up to almost any level you might need without problems. I do recall S3 having had an embarrassing outage or two over the last year, but the aggressive caching by CloudFront nodes should mean almost any S3 downtime would not affect CloudFront: only newly-uploaded content being accessed for the first time during an outage would hit an error. It should be possible to mirror content between CloudFront and another CDN using a different DNS setup

The future

S3Fox gave a good service initially, apart from the lack of support for adding headers to objects – I’ve heard there is a beta version now which has this feature, so I will revisit this software once the next version is released. In the mean time, S3Hub gives me this ability. I’ll also be looking at some way to automate uploading new content as part of a deployment script.

ESI support would also be nice, as would hot linking protection of some sort. It should be possible to achieve something similar using Javascript on the client end, getting even better performance by caching the static components on the client end after fetching them from the CDN, with the obvious drawback of normally needing some sort of fallback option for users without Javascript.

Backup Roundup (part 5) - Update and Conclusions

2009-01-11T00:00:00+00:00

Since writing the earlier parts of this mini-series, I discovered Mozy had managed to eat my backup entirely: had I lost any data in the last week or so and relied on Mozy to recover it, I would be in a whole world of data loss.

For the curious, here are the key log entries:

2009-01-03 12:01:45.387 MozyBackup[5632:4d0b] WRN (backup) Integrity check found problems in SQLiteManifestDatabase, attempting to recover 2009-01-03 12:01:45.388 MozyBackup[5632:4d0b] WRN (backup) —> * in database main * Page 24250: unable to get the page. error code=266 On tree page 24183 cell 190: Child page depth differs On tree page 24183 cell 191: Child page depth differs

After this point, Mozy proceeded to download a fresh ‘manifest’ from the server (nearly 200 Mb of it!) – then re-upload all my data, which has been underway for a week now and has over a week remaining (on a connection with 800kbps of upload capacity).

The PC version of Mozy has gained some UI features – in particular, a status indicator on each file (a green dot when the file is currently backed up, orange when it is awaiting backup), eliminating one distinction between it and Carbonite. No sign of this feature on the Mac yet, though.

The issue with Carbonite interfering with MSI software installations by accessing the temporary files (which seems to conflict with the locks Windows Installer needs) even though that folder is not set to be backed up remains a minor irritation – apparently Carbonite have a tool which applies some sort of workaround to prevent Carbonite accessing that area in the first place, but just remembering to suspend backups while installing new applications works fine. Unlike Mozy, suspending backups is a matter of two mouse clicks, with a third to resume backups once you’re finished. Usefully, if you forget to resume, this is done automatically after 24 hours.

Carbonite, on the other hand, is still going strong; Time Machine is also behaving itself perfectly, with the occasional check in Disk Utility to guard against the problem I had backing up over the network before. I’ll be replacing Mozy on my Mac as soon as Carbonite – or another product – is available. In the mean time, I have Time Machine as my backup solution and Mozy lurking there since I’ve paid for it for the time being.

I used Time Machine early in 2008 to migrate from one laptop to another: as soon as my new MacBook Pro arrived, I connected my USB backup drive and restored to the new system. All my settings, applications and everything else came across to the new machine in about an hour, without further intervention; as I recall, I had to reinstall or reactivate Adobe Creative Suite CS3 (presumably something to do with their licensing/activation system) and that was it. A bare-metal restore, to (slightly) different hardware, in an hour: backup doesn’t get much better than that! (There is some sort of transfer facility in Windows XP and Vista which I have used on occasion for similar purposes – nothing like as slick, however.)

The Never-Ending BT Battle

2009-01-09T00:00:00+00:00

In the small office I sometimes use, I have fairly typical requirements: one ordinary telephone line, plus ADSL Internet access. Not exactly rocket science, you would think. Unfortunately, once you get BT’s bureaucracy involved, even the simplest tasks become Herculean.

Initially, we had two very minor issues: we were on the basic tariff rather than the one we had requested (for businesses, BT offers ‘Business Plan’, where calls to landlines are capped at 10p for the first hour: quite a saving compared to the usual rate), and we wanted Caller ID and a few other features enabled. The extra features could be requested online, the tariff change required a phone call – and it all went downhill from there…

One small snag appeared in the online order: the confirmation message had the wrong address for “installation” of the features on our line (a street number, 1, had been inserted in our address). I called to query this, but was assured this didn’t matter: since the line was installed already and the features would be enabled from the exchange end without an engineer visiting, all that mattered was that the phone number was correct. So, I proceeded to deal with the tariff change request.

During that call, the operator suggested it would be cheaper to get a bundle, called “Featureline Compact”, since that gave all the features we wanted as standard. The only drawback – we thought – was that the new bundle required dialling 9 for all calls, since it’s designed as an alternative to operating your own internal exchange. Not a problem: I could program our own exchange to insert the prefix automatically anyway.

The one problem here is that changing to this bundle involved an internal BT procedure called a “cease and reprovide”: on paper, they end the old service then start providing a new one. In practice, this entails sending an engineer to the exchange to change the port settings, or moving from a standard port to one with Featureline support.

Instead, the line simply stopped working entirely. Upon calling BT Faults, we were told this was because we had requested that the line be “ceased”! Further investigation revealed that somehow, our “cease and reprovide” had somehow been split in half: only the “cease” request went through. Unfortunately, by this point it was too late to fix: our only option was apparently to request a new line installation.

A few days later, I was told that I had asked for DSL service to be cancelled as well. I hadn’t, of course – this was an automatic side-effect of the line being cancelled by mistake. Again, I was told it was too late to prevent the cancellation: I would have to request “new” service on the “new” line.

Next, two extremely large invoices (one for each service) arrived. Having only had the line a few months, we were well inside the one year minimum contract term – the cancellation meant paying the monthly fees for the rest of the year immediately.

There was light at the end of the tunnel, though: BT also contacted me to say they would be sending an engineer round – to the wrong address I mentioned earlier – the following Tuesday to install the “new” line. Quite why re-enabling an existing line required an engineer’s visit, I don’t know, but I would be spending Tuesday in another city in meetings all day anyway, so that wasn’t possible. I called and explained this, pointing out both the wrong address and the impossibility of their suggested date.

So, I returned to the office on Wednesday to find a note from the company next door (the owners of the incorrect address) that a BT engineer had visited, looking for “Mr or Mrs Sullivan”, despite having been told that this was the wrong address and that the date was not suitable.

Later that week, I was told BT had “found” a spare line which was listed as being at this incorrect address and activated it, rather than send another engineer out on another wild-goose chase. Fortunately, the line happened to be the right one, so we finally had a working phone line. (No Internet access yet: that would take another week to process.)

Throughout this process, I had been trying to contact BT Complaints in various ways. The online “chat” system was not working – around 11 o’clock one morning, it claimed to be closed since it is only open 8 am to 5 pm – and emails just produced multiple automatic acknowledgements and empty promises someone would call. (BT Faults were much more pro-active: one of my calls to them resulted in at least four different people returning the same call over the course of one morning!)

Finally, I stumbled across a reference to a secret, unlisted, high level complaints department for cases where the regular complaints department had failed. Clearly this fitted my case exactly. This number was answered quickly, by someone who actually had access to both the phone and Internet bills – during that same call, he was able to cancel all the extra penalty charges for early termination, the installation charges for the “new” service and the call charges we’d paid on the wrong tariff, and finally promise that someone from the local office would call back within two working days to see if the reconnection to the Internet could be expedited.

Two days later, I was indeed called back, then quickly reconnected.

Of course, there had to be one final snag: having a new account meant we were billed for the coming quarter’s Internet access – having already paid for service for half that period under the previous account. Not to mention being charged for an unsolicited “Internet security pack”. I called, and was quickly told both sums had been credited back, reducing the bill by rather more than 50%. However, two weeks later these credits had not shown up, so I tried email. Apparently there was no record of my previous call or the credits I was told had been applied, so I had to explain the situation again from square one.

Some weeks later, we received a “red” (final reminder) bill – for the original amount, with neither of the promised credits applied. A further long – and expensive, since these calls were to a premium rate 0845 number which pays the recipient per minute for incoming calls – call finally produced written confirmation of the credit for the unsolicited service. Apparently, though, it was not possible to credit this account for the period of service already paid on the previous one: instead, that would be refunded to the original (now closed) account, then the period would be paid for again through the new account instead.

We’re now waiting for the promised refund. Any bets whether it actually turns up, or goes the way of the first account credits? We’re also still not on the requested One Plan tariff (combining Internet and telephony on a single bill) – but given the months of bureaucracy and weeks without service the last request caused, I’m a little reluctant to try fixing that this time…

Backup roundup (part 4) - Apple Time Machine on Mac

2009-01-07T00:00:00+00:00

With the latest incarnation of Mac OS X, Apple incorporated a clever backup approach. The core approach is quite standard: copy all the files to another drive, then provide a client interface which will copy files back again.

Behind the scenes, on a local drive, your backups live in a folder named Backups.backupdb; each machine backed up gets its own folder (named after the machine name) within that. That in turn contains one folder for each backup, named in the form YYYY-MM-DD-NNNNNN, plus one symbolic link named Latest which points to the most recent backup.

Inside that, you will find a folder for each backed up volume, with the same name. At first glance, it looks as if every backup contains a complete copy of the volume as it was when the backup was made — in reality, however, most of these files are hard links to existing copies: each new backup adds only the files which have actually changed since the previous one, saving a great deal of disk space. (This is almost exactly the approach used by BackupPC for network backups, except that BackupPC uses symbolic links and compresses the files.)

This is very simple, robust and efficient: your backup files are simply files, in a standard Mac OS (HFS+) file system. (One interesting technical note here: where an entire directory is unchanged from one backup to the next, rather than hard linking the individual files, Time Machine hard links the directory itself, a facility of Leopard which probably surprises a lot of Unix purists. Ars Technica has an article giving a more detailed description of Time Machine innards for the curious.)

For some reason, Time Machine does not use the same approach when backing up to a network drive: instead, it creates a ‘sparsebundle’ file system image (a virtual drive, consisting of multiple 8 Mbyte segments) and mounts that to perform backups. It also identifies the backups by MAC address, rather than computer name. This approach gives two drawbacks: first, you can’t switch between local and network access to the same drive (or at least, this will give two separate sets of backups: you cannot back up locally then restore over the network, or vice versa), and secondly in practical usage I soon lost my backups to file system corruption within the remote disk image. (The corruption was serious enough that Disk Utility indicated the volume was not recoverable, meaning my backups were lost.) Performance was also extremely poor over the network compared to local access: far slower than the network connection would explain.

The performance and corruption issues may be an issue with using a standard USB disk attached to an Airport Extreme base station, as opposed to a dedicated Time Capsule appliance, but I would hesitate to recommend relying on Time Machine over the network at present. This is a shame: clearly, backing up over the network would be a lot more convenient than attaching a backup disk directly to my MacBook Pro. As a result, my present backup arrangements combine Mozy (backing up whenever I have an Internet connection) with Time Machine to a USB drive whenever it is connected. Others have experimented with backing up via Time Machine to iSCSI devices or network shares with mixed results.

In the longer term, I suspect Apple will be integrating the snapshot facilities of Sun’s ZFS with the Time Machine user interface, so you can get access to previous versions without the need for a second drive. This won’t guard against file system corruption, disk failure or loss or theft of the device, but is much more convenient for laptop users. I haven’t tried it, but it may even be possible to achieve this right now with the ADC ZFS support; it certainly won’t involve major changes to the innards of Time Machine. ZFS also has integral compression support, which would add compression to Time Machine’s backups without any extra effort.

Features and advantages

No subscription fee
Clever user interface
Unobtrusive
“Bare metal” restore facility: boot from Leopard install DVD and you can restore your entire system in one go!

Limitations and problems

Serious problems with corruption of network backups
Different formats between local and network
No compression: 1 Gb of data occupies 1 Gb of backup space

Summary

Use it, but with directly connected storage not a network device! Best used in conjunction with a remote backup solution such as Mozy (or Carbonite, once available) to get the best of both worlds: instant access to individual old files, with access to a complete remote backup for disaster recovery.

Backup roundup (part 3) - Carbonite on Windows

2009-01-06T00:00:00+00:00

Carbonite offers much the same type of service as Mozy, reviewed in the previous part of this series. Unlike Mozy, the service is presently limited to Windows systems, although Mac support is in development.

The inner structure seems more opaque than Mozy’s, from a superficial exploration, but the end result is similar.

The user interface is much more polished than Mozy’s: the developers clearly put substantially more effort into the appearance and integration. Where Mozy just gives a progress indicator, Carbonite adds a small colored dot to each backup-eligible file in Explorer, green when the file has been backed up, yellow when the file (or the content of a folder) has been modified since the last backup. There is also a context menu extension, allowing you to exclude files or folders or force an immediate backup of that data: much more useful than Mozy’s daily system-wide batch backups.

This is very user friendly: you can see at a glance if there is a current backup of any given file or folder, trigger an immediate backup after an important change, or exclude data you don’t need backed up. On the downside, Carbonite is more selective: video files will only be backed up after you explicitly request this for each folder containing video data.

In addition, backups are more current: Mozy performs a daily backup of all data, while Carbonite will trigger much more frequently when you create or alter files in a backed up area. In version 3.5, Carbonite added the ability to restore previous versions, another advantage over Mozy.

Restoring data to a new machine also features clever handling of user accounts, giving you the option of creating a new user account to match the one backed up or restoring into an existing user’s account (regardless of name). This is quite useful for moving from one machine to another.

The only drawback I have experienced is a conflict with installing software from MSI packages: Carbonite is a little bit too efficient and tries to back up the Windows Installer’s temporary files while they are in use, causing occasional conflicts. Suspending Carbonite backups for the duration of the installation is usually enough to get around this, although on one occasion (on Windows Vista Enterprise x64) I had to disable the Carbonite Windows service before an application would install successfully.

Features and advantages

Instant file/folder status indicator
Easier configuration of files to back up
Progress indication, easy suspend/resume
Access to previous versions of files

Limitations and problems

Lack of option to include all files
Conflict with MSI installation
No web access to backed up files

Summary

In every respect except web access to files and Mac support, Carbonite currently beats Mozy. For Windows users, it’s an excellent choice; on the Mac, the best choice seems to be the combination of Mozy and Time Machine, giving similar functionality with faster restores at greater expense. Once it’s released, I’ll examine the Mac version of Carbonite and revisit this conclusion; so far, I expect Carbonite and Time Machine to prove a winning combination, but Mozy could yet close the gap.

Backup roundup (part 2) - Mozy on Mac and Windows

2009-01-05T00:00:00+00:00

Mozy was the first online backup service I tried using seriously. The pricing seemed attractive – around $300 for three machines for two years, without storage limits – as did the Mac support.

My first impressions were positive: a relatively simple setup process, picking the directories to back up, then leaving the client to upload the data to their servers. Ultimately, I wanted my Mac home directory backed up excluding my Parallels virtual machine – at 40 Gb, this would take quite a few days, so I enabled backups a few Gb at a time until I had the selection I wanted). The Windows directory structure was a little more complex, but not difficult to accommodate within Mozy: after a few days, both my Vista and XP systems were backed up as well. The presence of a second drive where most of my data lived didn’t cause any issues either.

Internally, the backup algorithm appears to be based on RDiff deltas, tracking file status in an SQLite database. That should enable delta compression, uploading only the changed parts of each file after the initial backup. Running a backup consists of “offering” the server batches of files – up to 500 at a time, it seems – which could be deletions, new files or new versions of existing files. The server then asks for any necessary data to be uploaded, after which the client moves on to the next batch.

A nice, sane design, but I have encountered several problems in actual usage. On Windows XP, at some point something became corrupted; the best solution Mozy could come up with was to uninstall, delete some leftover files manually then reinstall, which is irritating, particularly since this resulted in having to reconfigure Mozy then re-process every file in the backup set.

I hit two other irritating glitches in the Mac version: early in 2009 the file state database appeared to become corrupted, so Mozy began the multi-day process of rebuilding it. A few months earlier, it stopped obeying the path exclusions I had set (essentially, my whole home directory minus my Parallels virtual machine) – and started trying to upload my entire 30 Gb virtual hard drive. Once a file has been backed up, Mozy only sends the changes – but there is no support for resuming a partial transfer. As a result, every attempt to upload that 30 Gb virtual drive failed part-way, then started again from scratch the next time. The Mozy support response was not to use exclusions, but to add only the folders I wanted backed up in their entirety.

The most serious issue, however, came when I performed a fairly large restore operation. Trying to restore around 13 Gb of data (photographs from my recent vacation in Boston and New York, in RAW format) ran overnight — then informed me at the end that “some” files had not been restored. Three of them, apparently, but the logs gave no indication which files had been missed or why!

The user interface is also quite limited and crude compared to Time Machine and Carbonite, although the Windows version recently started adding status icons to files in Explorer in the same way Carbonite does.

One significant advantage with Mozy comes in the web interface. Where Carbonite is built around an elaborate client package which handles everything, Mozy allows you to download backed up data from their website, wherever you are; Carbonite does not. This proved useful when I needed to access some data on my home system from elsewhere, and would also be very helpful in a disaster recovery situation (where the backed up system is unavailable) when you need urgent access to a particular part of the data: Carbonite requires you to install the client software, rather than just logging in to the website. Mozy also allows you to order, for a charge, copies of your data on optical media — with a reasonable Internet connection, though, you would have to be restoring several tens of gigabytes for this to be faster than the free option of restoring online.

Overall, the product is a good one, with some irritating glitches and rough edges. It is slowly improving; my only serious concern is the problem I encountered with a large file restore apparently missing a few files without detail or explanation. Barring significant improvements, though, I will be switching my two Windows systems from Mozy to Carbonite once the two years are up. Carbonite apparently have a Mac version in beta at present, with release scheduled early this year: I’m looking forward to comparing the two Mac versions when that happens.

Features and advantages

Simple, unobtrusive
Web access to your data
Status web accessible as well: see when all your systems were last backed up

Limitations and problems

Most recent version of files only: no backups of previous versions
Unexplained loss of all backup data on one occasion
Missed files during restore, without explanation
Inconsistent version numbering: the web site’s “1_2_0_3-3909” identifies itself as “1.2.0 (3-35603)” when installed

Summary

For Mac users, Mozy is a good choice, giving cost-effective off-site backup with a fairly trouble-free user experience, although not reliable enough to trust as a sole backup — best used as a fallback in the event Time Machine can’t help. For Windows, Carbonite is the better choice.

Backup roundup (part 1)

2009-01-04T00:00:00+00:00

No doubt we all know people who have lost data without having made backups — in my years of working for a big University, I’ve seen everything from a floppy disk apparently containing the sole copy of someone’s thesis to a hard drive containing gigabytes of course materials and email. For the first case, the answer is obvious: simply keeping a second copy on another disk, or on the network, would have avoided the problem entirely. For entire hard drives, the problem is far more difficult: even burning backups to DVD becomes cumbersome for this volume of data.

With gigabytes of data from digital photography, plus gigabytes more of stored email, version-controlled source code and various forms of content I work with for clients, data protection is a big challenge for me, and no doubt for many others in similar jobs.

A good starting point for hard drives is to move to a RAID array. My primary home storage is a RAID 5 array, my new office desktop is RAID 1. This protects me from data loss from the failure of any single hard drive, but gives no protection against file system bugs or accidental (or malicious) deletion. A more subtle vulnerability also exists: if a disk returns corrupted data, as opposed to returning an error, most implementations will accept the data without detecting any problem.

Sun’s ZFS and Network Appliance’s WAFL add checksums to detect disks returning corrupted data (as opposed to failing to return data, which is all a conventional parity-based RAID implementation will address). They also both offer snapshots, protecting against file deletion or alteration — but, as Joyent discovered with a week-long outage in early 2008, offering no protection against file system bugs.

At work, I was able to implement a network-based backup system for individual desktop machines: one Linux server with a large RAID 6 array running BackupPC which makes daily backups of all the client machines, using file level de-duplication and compression to pack backups of dozens of machines into a few cheap SATA hard drives. This has already saved several people’s bacon, pulling gigabytes of deleted files back in minutes. (ReiserFS issues have caused some downtime, so I have just switched to using ext3 instead.)

For typical (non-server) environments, there are two clear options: traditional backups to an extra local disk or optical media, or online backup services such as Mozy or Carbonite. Apple’s Time Machine is perhaps the most high profile example of the former, as well as being the most user-friendly. I’ve been using all three in a variety of settings for a while now, encountering the good and bad points of each. Over the next three posts, I intend to examine each in turn.

CDN experiences: good and bad, Amazon CloudFront vs CacheFly

2008-12-14T00:00:00+00:00

I’ve been building, hosting and maintaining a variety of websites, both commercially and personally, for over a decade now. In the early days, this involved a cutting-edge setup, migrating from the original CERN httpd to a new but rapidly spreading webserver called “Apache” (which had just broken through the 40% market share barrier), running on a veritable behemoth of a server named Dux: a Sun Ultra Enterprise 2, packing a pair of UltraSPARC processors at an awe-inspiring 167 MHz and 192 Mb of RAM — roughly equivalent to my iPhone today, but in those days a system that powerful would be serving dozens of interactive users simultaneously as well as hosting multiple websites.

The first site we installed under Apache was not the main University website, for a variety of reasons; instead, the first site was a mirror of Yale Medical School’s ‘GASNet’ Anesthesiology website for the benefit of UK users. In those days, transatlantic bandwidth was limited and expensive — Janet, the network linking UK universities to each other and to the Internet, had just introduced a system of charging universities for the transatlantic bandwidth, because the single 45 Mbit/sec link was becoming congested — so mirroring popular US websites was quite a useful thing to do.

A decade later, the connections are two or more orders of magnitude faster and Janet has stopped traffic charging again, but mirroring is still important for performance and reliability reasons. Large downloads benefit particularly, so file distribution sites like Kernel.org and SourceForge have geographically distributed mirrors, with the latter also using geolocation to steer users towards downloading from relatively local servers. Large commercial entities also mirror their sites, either setting up a system themselves or outsourcing the work to a CDN (Content Delivery Network) such as Akamai, who distribute the data across many servers and steer requests towards local servers. Akamai are an extremely large, successful and expensive example, with well over 10,000 servers across six continents: users will always get your data very quickly from a local server, wherever they may be, however busy your site may get — but you’ll pay a pricetag to match.

None of my projects have ever been in the market for a solution on Akamai’s scale, but I have always liked the idea of CDN hosting, both in terms of web performance — which was, after all, the subject of my final year dissertation as a student — and the improved efficiency of reducing unnecessary backbone traffic. So, when I saw the CacheFly CDN offering a free trial with a cheap package ($15/month), I jumped at the chance, quickly copying some frequently-accessed static content (images and CSS files) up to their servers and pointing my links there.

I was disappointed. At the time, I discovered my access — from the UK — was being routed to servers which appeared to be in Sweden, sending traffic on a bizarre tour of little-used Internet connections across mainland Europe which meant accessing my content through the CDN was actually slower than fetching from either of my own servers, even the one in California! (For obvious reasons, British web users visit American websites far more frequently than Swedish ones, so our ISP networks are designed accordingly: multiple, massive pipes across the Atlantic, plus a token amount of capacity which can get your packets to Sweden if you really insist.) Things seem to have improved slightly these days: UK users visiting CacheFly’s own website now seem to be served from Chicago rather than Sweden, which is probably a little faster. Still far from ideal!

Enter Amazon. Having discovered people were (mis)using their S3 online storage system as CDN-style static hosting, a task for which it was adequate despite not being intended or designed to function as a CDN, they took the next logical step and added a genuine CDN offering, CloudFront. Like S3, it was trivial to set up, with no commitment; unlike S3, it’s a genuine CDN. You upload data into an S3 “bucket” as before, but point users to a CloudFront hostname rather than an S3 one: Amazon’s system takes care of copying data from S3 into CloudFront servers for you as needed, as well as routing user requests to their nearest CloudFront server. In my case, this means London; of my two servers in San Jose, one is served from a node a mere 2 ms away, the other from St Louis.

I’ve moved all this site’s static content — the CSS, Javascript, images and downloads — to CloudFront delivery now, and the difference was immediately visible to me. The setup was trivial as well, once I installed the S3Fox Firefox plugin and found the relevant functions, first to set up an S3 bucket to hold my static content, then to associated a CloudFront ‘distribution’ with that bucket. Since there was a bit of trial and error involved, I’ll post the relevant instructions soon. (I tried setting the CloudFront distribution up from the command line using Amazon’s documentation, but gave up on getting the CNAME associated and used S3fox instead). Right now, I have two frustrations: first, the S3Fox plugin doesn’t appear to set permissions when uploading, so I have to fix that manually; secondly, it doesn’t set Expires headers on files, although this feature is apparently added in the next version.