Sunday, November 18, 2012

Creating the Tribblix ramdisk

When you're running Tribblix off the live iso image, most of what you're using is actually just one file - the initial ramdisk loaded into memory.

Putting together the ramdisk was one of the trickier areas of getting Tribblix working. It tok a while to work out exactly what needed to be in there.

As part of the build, a minimalist OS is installed into a build area. The simplest approach is to put all of that into the ramdisk. That works, but can be pretty large - for a base build, you're looking at a 512M ramdisk. While this is fine for many modern systems, it's a significant constraint when installing into VirtualBox (because you can only assign a relatively small fraction of your available memory to the entire virtual instance). Besides, being efficient is a target for Tribblix.

So what happens is that /usr, which is the largest part, and can get very large indeed, is handled separately. What ends up in the ramdisk is everything else, with /usr mounted later.

However, there's a catch. There's a tiny amount of /usr that needs to be in the ramdisk to get /usr mounted. Part of this is intrinsic to the special mechanism that's used to mount /usr, and it took some experimentation to work out exactly what files are required.

Other than /usr, the ramdisk contains everything that would be installed. The installation routine simply copies the running OS to disk (and then optionally adds further packages). So there's no fiddling around with what's on the ramdisk. (In OpenSolaris and OpenIndiana, some of the files are parked off in solarismisc.zlib and linked to. I don't need to do that, so solarismisc.zlib doesn't exist in Tribblix.)

And because the contents of the installed system are taken straight off the ramdisk, the ramdisk contains both 32 and 64-bit files. Creating separate 32 and 64-bit ramdisks might make each ramdisk smaller, but would take up more space overall (because there is duplication) and makes the install much more complex. Thus, when grub boots, it uses $ISADIR to choose the right kernel but the boot archive is fixed.

So how is the ramdisk built? It's actually very simple.
  1. Use mkfile to create a file of the correct size, such as 192m
  2. Use lofiadm to create a device containing the file
  3. Use newfs to create a ufs file system on the device. Because we know exactly what it's for we can tune the free space to zero and the number of inodes
  4. Mount that somewhere temporarily
  5. Copy all the temporary install location to it, except /usr
  6. Copy the handful of files from /usr into place
  7. Drop an SMF repository into place. (I copy one from a booted system that's correctly imported.)
  8. There are a few files and directories need by the live boot that need to be created
  9. Unmount the file system and remove the lofi device, then gzip the file.
  10. Then copy the compressed file into where you've told grub to look for the boot archive (/platform/i86pc/boot_archive)
A corollorary to this design is that the iso doesn't need to contain copies of what's in the ramdisk - the iso itself just has /platform (to boot, and containing the ramdisk), solaris.zlib that contains /usr, pkgs.zlib that contains additional packages, and a few odd files required by the installation script.

Tuesday, November 06, 2012

Tribblix Milestone 1

An updated release of Tribblix is now available for download.

This version is built from the same base as the initial release, namely OpenIndiana OI151a7, so is whatever version of Illumos that corresponds to.

Milestone 1 adds Xfce, Firefox 16.0.1, emacs, python, cmake, and AfterStep.

Despite all the additional software, the ISO image isn't much larger than before, due to significant improvements in the way that the ISO is constructed. Essentially, there was quite a bit of duplication on the ISO of files that were already in the ramdisk.

Tribblix is still a long way from production ready, but for everyday office use it's got a decent desktop, an editor, and a web browser, so that's a fair fraction of the workload covered.

Monday, October 29, 2012

How to build Tribblix

The scripts and configuration files that I used to build Tribblix are now available on github.

There are currently two components available. There are some more bits and pieces that I'm still working on. (Specifically, the live image manifests and method scripts, and the overlay mechanism.)

First, the ips2svr4 repo contains the scripts used to create the SVR4 packages. The initial Tribblix prerelease was based on an OpenIndiana 151a7 install, I simply converted all the packages wholesale. One of the problems I had was that an installed system has had lots of configuration applied, so I needed to nullify many of the changes made to system files. There's also an equivalent script that can make SVR4 packages from an IPS on-disk repo; I've tested that I can make packages from an Illumos build, but haven't yet tried to build a system from them. (At least I know the OI binaries work on the system I'm testing.)

Next, the tribblix-build repo contains the scripts used to install the packages to a staging area, fix up that install, create the zlib files, build the boot archive, and create the iso, along with the live_install script that's used to install to hard disk.

The scripts are ugly. Mostly, I was doing the steps by hand and simply saved the commands into the scripts so I didn't have to type it next time. That they can be improved is undoubted. I hope I've taken most of the profane language in the comments I put in as I found out what worked and what didn't along the way.

The second problem anybody else is going to have with the scripts is that they have myself embedded in them. When I'm doing the construction, I'm always working from my own home directory, so it's currently hard-coded, as are all the other locations. That will get fixed, but it's more fun building a better distro than making the scripts suitable for a beauty contest.

Friday, October 26, 2012

Those strange zlib files

If you look at an OpenIndiana live CD, you'll see a couple of strangely named files - solaris.zlib and solarismisc.zlib. What are these, how can you access their contents, and how can you build your own?

These are actually archives of parts of the filesystem. In the case of solaris.zlib, it's /usr; for solarismisc.zlib it's /etc, /var, and /opt.

These are regular iso images, compressed with lofiadm. So you can use lofiadm against the file to create a lofi device, and mount that up as you would anything else. During the live boot, solaris.zlib gets mounted up at /usr. (There have to be some minimal bits of /usr in the boot archive; that's another story.)

Building these archives is very easy. Just go to where the directory you want to archive is and use mkisofs, like so:

mkisofs -o solaris.zlib -quiet -N \
    -l -R -U -allow-multidot \

    -no-iso-translate -cache-inodes \
    -d -D -V "compress" usr


Then you can ask lofiadm to compress the archive

lofiadm -C gzip solaris.zlib

The compression options are gzip, gzip-9, and lzma.  On the content I'm using, I found that basic gzip gives me 3-4x compression, and lzma a bit more at 4-5x. However, using lzma takes an order of magnitude longer to compress, and you get maybe half the read performance. There's a trade-off of additional space saving against the performance hit, which is clearly something you need to consider.

For Tribblix, I use the same trick, although I'm planning to get rid of the extra solarismisc.zlib. And I'm hoping that being lean will mean that I don't need the extra lzma compression, so I can stay fast.

Wednesday, October 24, 2012

Building Tribblix

I've recently been working on putting together a distribution based on OpenSolaris, OpenIndiana, and Illumos.

Called Tribblix, it was quite challenging to put together. I'll cover the individual pieces in more detail as time goes on, and put the code up so anybody else can do the same. But here's the rough overview of the process.

First, I've created SVR4 packages - I can do this either from an installed system, or from an on-disk repo. The IPS manifests contain all the information required to construct a package in other formats, not only what files are in the package, but also the scripting metadata that can be used to generate the SVR4 installation scripts. The most complex piece is actually generating the somewhat arcane and meaningless SVR4 package name from the equally arcane and meaningless IPS package name, making sure it fits into the 32 character limit imposed by the SVR4 tools. This has to be repeatable, as I use the same transformation in dependencies.

(An aside on dependencies: there are significant areas of brokenness in the IPS package dependencies on OpenIndiana, not to mention problems with how files are split into packages. There's significant refactoring required to restore sanity.)

Then I simply install the required packages for a minimal system into a build area. Of course, it took a little experimentation to work out what packages are necessary. There's an extra package for the OpenSolaris live CD that goes on as well.

There's then some fiddling with the installed image, setting up an initial SMF repository, SMF profiles, and grub. Something in need of more attention is the construction of the boot archive. I've got it to work, but it's not perfect.

The zlib archives you find on the live cd are then put together (these are lofi compressed iso images), plus an extra archive containing extra packages, and then mkisofs creates the iso.

The installer is very simple. I need to work on automating disk partitioning, as it's currently left to the user to configure the disk by hand. Then a ZFS pool is created and mounted up. The booted image is fairly minimalist, so that is simply copied across wholesale. It's unlikely that you would want less on the installed system than you would on the initial boot. I delete the live cd package, and then optionally add sets of additional packages.

Then grub configuration, setting up SMF for a real boot (which has different SMF profiles), creating the real boot archive (which currently takes an order of magnitude longer than it should), and finally setting up the live filesystems so they end up in the right place.

That's pretty much it. Descriptions of each step will be forthcoming in future blog posts, together with the code on github.

Tuesday, September 25, 2012

Mangling the contents file for fun and profit

I was recently using Live Upgrade to update a Solaris 10 system, when it went and chucked the following error at me and refused to go any further:

WARNING: Directory </usr/openv> zone <global> lies on a filesystem shared between BEs, remapping path to </usr/openv-S10U10>.
WARNING: Device <storage/backup> is shared between BEs, remapping to <storage/backup-S10U10>.
Mounting ABE <S10U10>.
ERROR: error retrieving mountpoint source for dataset <storage/backup-S10U10>
ERROR: failed to mount file system <storage/backup-S10U10> on </.alt.tmp.b-yPg.mnt/usr/openv-S10U10>

What I have is Netbackup installed in /usr/openv, and this is a separate filesystem (in fact, it's on a completely separate pool on completely separate drives).

The underlying problem here is that Solaris thinks there's part of the OS installed in /usr/openv, so it gets included in the scope of the upgrade. I've seen this in other cases where someone has stuck Netbackup off to the side, and it drags way too much stuff into scope.

On backup clients, the simplest thing to do is remove the client and reinstall it when you're done. You might get a free upgrade of the backup client along the way.

In this case I was working on the master backup server and didn't want to touch the installation at all.

One advantage of SVR4 packaging is that the package "database" is just a bunch of text files. If anything gets messed up you can open them up in your favourite editor and fix things up. (In one case, I remember simply copying /var/sadm off a similar system and nothing noticed the difference.)

The list of what files are installed is the contents file (found at /var/sadm/install/contents). The Live Upgrade process looks at this file to work out where software is installed. So all I had to do here was

grep /usr/openv contents > contents.openv

to save the entries for later, and

grep -v /usr/openv contents > contents.new
mv  contents.new contents

to create a contents file without any references to /usr/openv.

After this, Live Upgrade worked a treat and didn't try and stick its nose where it wasn't wanted.

Then, after the upgrade, I just had to merge the saved contents.openv I saved above into the new contents file. There's a bit of a trick here - the contents file is sorted on filename, so you can't just cat them together. I opened up the new contents file, went to the right place, inserted the saved entries, and was good to go.

Recursive zfs send and receive

I normally keep my ZFS filesystem hierarchy simple, so that the filesystem boundary is the boundary of administrative activity. So migrating a filesystem from one place to another is usually as simple as:

zfs snapshot tank/a@copy
zfs send tank/a@copy | zfs recv cistern/a

However, if you have child filesystems, and clones in particular, that you want to move, then it's slightly more involved. Suppose you have the following

tank/a/myfiles
tank/a/myfiles@clone
tank/a/myfiles-clone

where myfiles-clone is a clone of the @clone snapshot. I often create these temporarily if someone wants a copy of some data in a slightly different layout. In today's case, it had taken some time to shuffle the files around in the clone and I didn't want to have to do that all over again.

So, ZFS has recursive send and receive. The first thing I learnt is that myfiles-clone isn't really a descendant of myfiles - think of it as more of a sibling. So in this case you start from tank/a and send everything under that. First create a recursive snapshot:

zfs snapshot -r tank/a@copy

Then to send the whole lot

zfs send -R tank/a@copy

I was rather naive and thought that

zfs send -R tank/a@copy | zfs recv cistern/a

would do what I wanted - simply drop everything into cistern/a, and was somewhat surprised that this doesn't work. Particularly as ZFS almost always just works and does what you expect.

What went wrong? What I think happens here is that the recursive copy effectively does:

zfs send tank/a@copy | zfs recv cistern/a
zfs send tank/a/myfiles@copy | zfs recv cistern/a
zfs send tank/a/myfiles-clone@copy | zfs recv cistern/a

and attempts to put all the child filesystems in the same place, which fails rather badly. (You can see the hierarchy that would be created on the receiving side by using 'zfs recv -vn'.)

The way to solve this is to use the -e or -d options of zfs recv, like so:

zfs send -R tank/a@copy | zfs recv -d cistern/a

or

zfs send -R tank/a@copy | zfs recv -e cistern/a

In both cases it uses the name of the source dataset to construct the name at the destination, so it will lay it out properly.

The difference (read the man page) is that -e simply uses the last part of the source name at the destination. In this example, this was fine, but if you start off with a hierarchy it will get flattened (and you could potentially have naming collisions). With -d, it will just strip off the beginning (in this case, tank), so that the structure of the hierarchy is preserved, although you may end up with extra levels at the destination. If it's not quite right, though, zfs rename can sort it all out.

Monday, September 10, 2012

iTribble

Just over a month or two ago, I wasn't an Apple customer. Sure, my daughter had an iPad (which I had used for a little development), but I didn't own or use any Apple devices myself.

Then my company mobile phone gave up the ghost. I had a Galaxy SII, and was assuming that I would get an SIII when the renewal came due. However, my old phone simply died a few days before the SIII became available in the UK, and I got an iPhone instead as it was actually available there and then.

The iPhone is OK, I guess. I'm not really a heavy smartphone user, it's handy to have some of the features but I wouldn't say that they're really crucial to me. My own personal phone is many years old now, and is one of those increasingly rare device that's actually useful for making phone calls. Generally, though, I wouldn't say that the iPhone is dramatically better than the Galaxy SII I had before; it's a bit more polished, but that's about all.

When it comes to spending my own cash, I then got myself a new iPad. The one with the retina display. I had been meaning to for a while, but was often too busy.

I had used several iPads before, and they've always just felt right. The touch, weight, balance, quality, all combine to generate an excellent experience. I wanted something around the home with reasonable battery life, instant on, and something that doesn't need a magnifying glass or too precise finger location. And the iPad delivers.

The main thing I use it for, a lot, is Sky Go. Generally in catchup mode, rather than live. (It's one of those odd things. Of all the programming that's available, a small fraction is what I want to watch. Invariably there's a multiway conflict, followed by hours or days of total wasteland.)

Generally, I find the Sky Go player to work extremely well. Better than iPlayer, anyway (whether that's the player or the delivery mechanism, though, I'm not quite sure). But then I notice that the latest release of the iPlayer app can download content to the iPad for viewing later, which may come in extremely useful.

I got a little iPod shuffle along with it. It's just great. I had an mp3 player from a brand that I had never heard of, and it wasn't reliable, nor did it have reasonable battery life. The whole thing put me off. But I wanted a little distraction at the gym, so the shuffle was perfect - you don't want to look at it or fiddle with the controls, ever, beyond on and off, and I wanted the smallest and lightest model available. I find myself using it so much now that I could actually do with a larger capacity model to get more tracks in the mix.

The latest toy is a new MacBook Pro. A new company laptop was due, I'm known to be a unix guy, so got offered a choice. My only constraint was that the resolution be adequate. (Seriously, 1366x768 is so 1990s.) So the retina display was called for again.

Frankly, I love it. So the keyboard is different, the trackpad is different, the user interface is different. But I quickly became accustomed to it, especially when things actually work. Like the iPad, though, the user experience is dramatically superior. And that old Windows thing from a mainstream supplier I had before is utter garbage in comparison.

Unfortunately I'm having a bit of trouble persuading the company to go for the dual thunderbolt 27-inch display setup.

So, largely by accident, and certainly without deliberate planning, most of my devices happen to have an Apple logo on.

Sunday, September 02, 2012

Cargo Cult IT

In a Cargo Cult, practitioners slavishly imitate the superficial behaviours of a more advanced culture in the hope that they will receive the benefits of that more advanced culture.

I'm seeing signs that Cargo Cult behaviour is becoming prevalent in IT. Some examples that come to mind are agile, cloud, and devops.

This isn't to say that these technologies are inherently flawed. Rather, just as in the true Cargo Cults, adherents completely miss the point and hope to reap the benefits of a technology by blindly applying its superficial manifestations in formulaic fashion.

Let's be absolutely clear - many advanced organizations are using agile, cloud, devops, and other technologies to great effect. The problem comes when more primitive societies merely emulate the formalism without any clear understanding of the reasons behind it - or even an acceptance that there are underlying reasons.

Slavishly imitating the behavioural patterns of a more successful organization is unlikely to lead to a successful outcome. Rather, understanding your own organization's problems and then understanding how other organizations have attacked theirs, and why they have adopted the solutions they have, will allow progress.


Of course, there are other cults that are simply false. I'm tempted to drop ITIL and ISO9000 straight into that bucket.

Monday, June 04, 2012

JKstat in Javascript

I've just released version 0.70 of JKstat, which brings in a couple of new features that I've had sitting off to the side for a while.

The simplest is an implementation of a RESTful server using Jersey. This is just a handful of annotated classes and an updated build script to create a war file that can be dropped into tomcat. On its own, this isn't terribly interesting - the JKstat client can use the XML-RPC interface just fine, and that's easier to implement.

However, there are a number of client interfaces that are much easier to get working if you're using RESTful interfaces. One is the java applet version of the JKstat browser; another is anything using javascript on the client.

Which leads me to the other new feature here. Spurred on by Mike Harsch's mpstat demo, I've put together a very simple browser based client for JKstat, using javascript.


It's just a prototype, really. But it has the basic interface features you would need. On the left the kstats are arranged in a hierarchy, thanks to jsTree. In the main panel is a continuously updating table of the statistics and their values and rates, and above is a graph of the data implemented using Flot.

Sunday, May 27, 2012

Lessons learnt from serving Queen Victoria's Journals

Towards the end of last year I was asked about how easy it would be to launch a very public website. Most of what the company does is relatively highly specialised, low traffic, high value, for a very narrow and targeted audience.

We were essentially unfamiliar with sites that were wide open and potentially interesting to the whole world (or a large fraction thereof). And we knew that there would be widespread media coverage. So there were real concerns that whatever we built would buckle, turning into a PR disaster. (Everyone's heard of the census launch, I expect.)

Almost 6 months later, we launched Queen Victoria's Journals. And yes, it ended up both nationally and locally on the BBC, in the UK newspapers including The Guardian, The Independent and the Daily Mail, and overseas in Canada and India.

After a huge amount of work, the launch went without a hitch. Traffic levels were right where we expected, and the system handled the traffic exactly as predicted. What's also clear is that if we hadn't done all the preparation work, it would most likely have been a disaster.

We're using pretty standard components - Java, Apache, Tomcat, Solr - and as I've explained previously, we maintain our own software stack. This is all hosted on Solaris Zones, built our way - so we can trivially build a bunch more, clone and restore them.

There's no real tuning involved in the standard components. They'll cope just fine, provided you don't do anything spectacularly stupid with the applications or data that you're serving. I built an isolated test setup, cloned regularly from a development build, so that I could run capacity tests without my work being affected by or impacting on regular development.

The site doesn't have that many pages, so I started by simply testing each one - using wget or ab (apache bench). I needed a whole bunch of servers to send the requests from - easy, just build a bunch more zones. And this showed that we could serve hundreds of pages a second from each tomcat, apart from one page which was returning a page every few seconds. The problem page - it's the Illustrations page linked to from the main toolbar - was being created dynamically via multiple queries to the search back-end which were being rendered each time. The content never changes (until we update the product, at any rate) so this is really a static page. Replacing it with something static not only fixed that problem, but dramatically reduced the memory footprint of tomcat, as we were holding search references open in the user session and generating huge numbers of temporary objects each time it was rendered.

The server capacity and performance issues solved, we went back to looking at network utilization. That's harder to solve from an infrastructure point of view - while I can trivially deploy a whole bunch more zones in a minute or so, it takes months to get additional fibre put in the ground. And our initial estimates, which were based on the bandwidth characteristics of some our existing sites, indicated we could well get close to saturating our network.

The truth is, though, that most sites are pretty inefficient, and ours started out as no exception. We got massive wins from compressing html with mod_gzip, we started to minify our javascript, and were able to dramatically decrease the file size of most of the images. (Sane jpeg quality settings are good; not including a 3k colour profile with a 10 byte png icon also helps.) Not only did this decrease our bandwidth requirements by a factor of 5 or more, it also improves responsiveness of the site because users have to download far less.

Most of the testing for bandwidth was really simple - construct a sample test, run it, and count the bytes transferred by looking at the apache logs. Simply replaying the session allows you to see what effect a change has, and you can easily see which requests are most important to address.

We also took the precaution of having some of the site hosted elsewhere, thanks to our good friends at EveryCity. They're using a Solaris derivative so everything's incredibly simple and familiar, making setup a breeze.

We learnt a lot from this exercise, but one of the primary lessons is that building sites that work well isn't hard, it just requires you not to do things that are phenomenally stupid (taking several seconds to dynamically generate a static page) or obviously inefficient (jpeg thumbnails that are hundreds of kilobytes each), that javascript minifies very well, and html (especially the hideously inefficient html I was looking at) compresses down really well.

Test. Identify worst offender. Fix. Repeat. Every time you go round the loop improves your chances of success.



Wednesday, May 23, 2012

Simple Zone Architecture

I use Solaris zones extensively - the assumption is that everything a user or application sees is a zone, everything is run in zones by default.

(System-level infrastructure doesn't, but that's basically NFS and nameservers. Everything else, just build another zone.)

After a lot of experience building and deploying zones, I've settled on what is basically a standard build. For new builds, that is; legacy replacement is a whole different ballgame.

First, start with a sparse-root zone. Apart from being efficient, this makes the OS read-only. Which basically means that there's no mystery meat, the zone is guaranteed to tbe the same as the host, and all zones are identical. Users in the zone can't change the system at all; which means that the OS administrator (me) can reliably assume that the OS is disposable.

Second, define one place for applications to be. It doesn't really matter what that is. Not being likely to conflict with anything else out there is good. Something in /opt is probably a good idea. We used to have this vary, so that different types of application used different names. But now we insist on /opt/company_name and every system looks the same. (That's the theory - some applications get really fussy and insist on being installed in one specific place, but that's actually fairly rare.)

This one location is a separate zfs filesystem loopback mounted from the global zone. Note that it's just mounted, not delegated - all storage management is done in the global zone.

Then, install everything you need in that one place. And we manage our own stack so that we don't have unnecessary dependencies on what comes with the OS, making the OS installation even more disposable.

We actually have a standard layout we use: install the components at the top-level, such as /opt/company_name/apache, which is root-owned and read-only, and then use /opt/company_name/project_name/apache as the server root. Similar trick works for most applications; languages and interpreters go at the top-level and users can't write to them. This is yet another layer of separation, allowing me to upgrade or replace an application or interpreter safely (and roll it back safely as well).

This means that if we want to back up a system, all we need is /opt/company_name and /var/svc/manifest/site to pick up the SMF manifests (and the SSH keys in /etc/ssh if we want to capture the system identity). That's back up. Restore is just unpacking the archive thus created, including the ssh keys; cloning a system you just unpack a backup of the system you want to reproduce. I have a handful of base backups so I can create a server of a given type from a bare zone in a matter of seconds.

(Because the golden location is its own zfs filesystem, you can use zfs send and receive to do the copy. For normal applications it's probably not worth it; for databases it's pretty valuable. A limitation here is that you can't go to an older zfs version.)

It's so simple there's just a couple of scripts - one to build a zone from a template, another to install the application stack (or restore a backup) that you want, with no need for any fancy automation.

Sunday, May 20, 2012

Vendor Stack vs build your own

Operating System distributions are getting ever more bloated, including more and more packages. While this reduces the need for the end user to build their own software, does it actually eliminate the need for systems administrators to manage the software on their systems?

I would argue that in many cases having software you rely on as part of the operating system is actually a hindrance rather than a help.

For example, much of my work involves building web servers. These include Java, Apache, Tomcat, MySQL and the like. And, when we deploy systems, we explicitly use our own private copies of each component in the stack.

This is a deliberate choice. And there are several reasons behind it.

For one, it ensures that we have exactly the build time options and, (in the case of apache) the modules we need. Often we require slightly different choices than the defaults.

Keeping everything separate insulates us from vendor changes - we're completely unaffected by a vendor applying a harmful patch, or from "upgrading" to a newer version of the components.

A corollary to this is that we can patch and update the OS on our servers with much more freedom, as we don't have to worry about the effect on our application stack at all. It goes the other way - we can update the components in our stack without having to touch the OS.

It also means that we can move applications between systems with different patch levels, able to go to both newer and older systems easily - and indeed, between different operating systems and chip architectures.

As we use Solaris zones extensively, this also allows us to have different zones with the components at different revision levels.

With all this, we simply don't need a vendor to supply the various components of the stack. If the OS needs them for something else then fine, we just don't want to get involved. In some cases (databases are the prime example) we go to some effort to make sure they don't get installed at all, because some poor user using the wrong version is likely to get hurt.

All this makes me wonder why operating system vendors bother with maintaining central copies of software that are no use to us. Indeed, many application stacks on unix systems come with their own private copies of the components they need, for exactly the reasons I outlined above. (I've lost count of the number of times something from Sun installed it's own private copy of Java.)

(While the above considers one particular aspect of servers, it's equally true of desktops. Perhaps even more so, as many operating system releases are primarily defined by how incompatible their user interface is to previous releases.)

Saturday, March 17, 2012

Does anybody still use Java applets?

It's been an awful long time since I thought about Java applets.

A while ago I updated a Java jigsaw application, Sphaero 2, and it was just a regular desktop application. Recently I got asked if it was possible to use it in a web page - as an applet.

This turned out to be incredibly easy. In parallel with the main application (which is a JFrame), implement the same code as part of a JApplet. Took me a few minutes to do (and a bit longer to clean it up and refactor it), so if you go to the Sphaero 2 page there are now a number of sample images that will launch a Java applet if you've got Java support in your browser.

Encouraged by this, I added applet support to JKstat. The idea is to set up a JKstat server, and if you point a web browser directly at the server then you get a page containing the JKstat applet, which can then connect back to the server it was downloaded from (the applet security model says that you can connect back to where you came from, so that's OK) to gather statistics.

This proved to be a little harder. It looks like this only works (in the case of an unsigned applet) for the REST variant of the client-server protocol. If I try it with XML-RPC then it looks like it wants to get DTDs for validation, and gets security exceptions trying to get them. So the standalone JKstat server doesn't work as is.

But that's not too bad, because I've got several options for serving the data using the REST protocol. From my sample play server, to node-kstat, or I've been testing a RESTful server based on Jersey.

It's been a useful learning exercise, and I actually find the results to be quite useful. Maybe applets will become fashionable again?