Metacomputing

Toby Howard

This article first appeared in Personal Computer World magazine, June 1998.

JUST OVER A YEAR AGO, Futures reported on a huge parallel supercomputer called TFlops, then undergoing installation at the Sandia National Laboratories in New Mexico. Now in full operation, it's the most powerful supercomputer on Earth. A gargantuan ensemble of over 9000 interconnected Pentium Pros, it routinely clocks over 1.3 million Mhz, occupies a huge room, requires full-time supervision, and consumes the same amount of power as a small town. But it may be the last of a dying breed. Now there's the prospect of new kind of massively parallel supercomputer which may have capabilities beyond our wildest imaginations. And you could be part of it.

The idea is to make use of the vast computational power wasted every day while thousands of PCs around the world aren't doing anything. While we sleep or take coffee breaks, our machines mostly sit idle, or run screensavers whose processing requirements have little impact on today's powerful CPUs. With an estimated thirty million machines on the Web, that's a lot of virtual power. If it were harnessed, the Web could become a gigantic distributed computer -- a "metacomputer".

Metacomputing is widespread in research laboratories, where programs with huge memory and processing requirements are carved up between local computers, whose individual results are stitched together to form the overall solution. In the lab, where each machine has access to a shared file system and a local high-speed network, metacomputing is easy. On the Web, however, organising metacomputing is far more difficult.

Difficult, but not impossible. First, the problem to be solved is split into thousands of smaller, more manageable problems, each of which is allocated to a volunteer. Using software and data downloaded from a "Problem Server", each small-scale problem can be solved off-line, and its results returned to the Problem Server, which acts as a computational clearing house.

One of the first metacomputing projects was GIMPS -- the Great Internet Mersenne Prime Search. A number is prime if it can only be divided by itself and 1. For example, the first few primes are 2, 3, 5, 7, 11, 13, 17, 19, 23 and 29. Although Euclid proved that there are infinitely many prime numbers, there is no known method for generating them -- you have to search for them.

Certain kinds of number are more likely to be prime than others, such as numbers of the form 2^N - 1, where N is prime. Primes of this kind are called Mersenne primes, after the 17th century French monk who first described them. Mersenne knew 11 such numbers, the largest being 2²⁵⁷ - 1. Now we know 37 of them, and GIMPS recently discovered the latest two. The current champion, and the largest prime number known, is 2^3021377 - 1, which if written out in full would need over 900,000 digits. Its discoverer was Ronald Clarkson, a student at California State University.

There are over 2,000 GIMPS volunteers worldwide, each of whom is willing to surrender some of their CPU time to the search. GIMPers are given specific numbers to test for primality. When I joined, for example, I was allocated 2^4745401 - 1, and my 200 MHz Pentium will take about 20 days of background computation to check if it's prime or not. Since the software runs at the lowest priority, it only steals CPU cycles when you're not doing something more important, like playing Quake 2.

Another successful metacomputing project tries to crack codes -- but it's perfectly legal. In order to demonstrate the security (or otherwise) of various encryption schemes, the American company RSA Data Security has published a number of cryptographic puzzles, offering prizes for their solution. In February this year, a team of 22,000 volunteers with access to 50,000 CPUs solved the problem in 39 days . A year ago a similar puzzle took twice as long to crack -- evidence that metacomputing is maturing.

Perhaps the most exciting application of metacomputing is the SETI@home project. The brainchild of Dan Werthimer of the University of California, SETI@home is designed to answer the biggest of Big Questions: is there life out there?

Since the 1970s, the Aricebo Observatory in Puerto Rico has hosted a project to hunt for intelligent signals hidden in the cacophany of deep-space radio waves. But the specialised search equipment, which works in real-time, can check only a tiny set of wavelengths for signals which might be intergalactic hellos. This is a problem tailor-made for metacomputing. Recordings of the radio noise will be placed on a Problem Server. Volunteers will download chunks to process with their SETI@home analysis software, which will run, ironically, as a screensaver, displaying information about the analysis. If ET is out there and broadcasting, it's a great way to try and listen. Alas, although technically sound and all ready to go, SETI@home is on hold due to lack of funds.

Web metacomputing has the potential to be the great democratiser of science. In many fields, discoveries would no longer be confined to computing labs where experts tend expensive monsters. The drudgery of scientific calculation could be shared out among the world's PCs. Everyone would get the chance to make their mark, solving problems, discovering new results, and perhaps even finding signals from an alien civilisation. It'll be a high-tech lottery, but like that bloke on the telly says, "It could be you".

Toby Howard teaches at the University of Manchester.