Turning 30 million old photos into digital gold

The image is black and white, but the picture is reassuringly familiar.

Hedgerows intersect with roads and paths at obtuse angles to create the recognisable patchwork of the British countryside. Houses, sheds and trees dot the fields. A quarry hole shimmers silver.

The next photo is the same, but different. The lines are consistent, but a copse of trees is gone. The spot-the-difference continues in the next image, this time it’s the outbuildings that have vanished.

And then, suddenly, everything is in colour. A new housing development has appeared in the bottom left of the frame.

Another jolt. Warehouses are everywhere in luminous technicolour.

Valuable data in the archives

This series of aerial images chart the shifting character of one corner of East Kilbride, taken at intervals between 1946 and 1988.

They, along with more than 30 million others from across the world, belong to the National Collection of Aerial Photography (NCAP), based in Edinburgh. The collection is rich with nostalgia, providing a god’s-eye view on how – and where – we used to live.

But it is richer still in other ways. It is a vast treasure trove of high-resolution data capable of tracking climate change, uncovering unexploded bombs, reheating cold cases for the police, and helping planners make better decisions for economy-boosting land development.

However, there is a problem. To unlock the archive’s potential in detecting munitions, cracking cases or boosting GDP the photos need to be digitised. To help, NCAP is working with experts at the University of Edinburgh to create novel techniques to convert crumbling celluloid into valuable data.

Declassified material

The scale of the task is as daunting as the collection is inspiring.

NCAP’s archive includes photographs from the pioneers of aerial photography in the 1920s to the vast military collections produced during the Second World War and the Cold War, as well as commercial photography of places around the British Isles and Commonwealth. The collection continues to grow as the United Kingdom’s Ministry of Defence declassifies more military-related aerial photography.

NCAP was established in the early 1960s with the role to collect and preserve the future of these records and to make them as accessible and available as possible. As the large majority of the collection is in physical ‘analogue’ format, and some of it in need of careful preservation and storage, NCAP has been running a programme to digitise all the items in its care.

The collection is of interest to many sectors, from documentary film makers and computer software companies, to researchers with an interest in land use change. It has even been used by Police Scotland when reviewing cold cases and in long-running investigations.

Boom time

Its largest customer base comes from the Explosive Ordnance Disposal sector in Europe. Its Second World War post-strike aerial photographs are vital to pinpoint the entry point of unexploded bombs in the ground and this is essential information for land developers in their risk assessments.

NCAP Head, Allan Williams, says: “The collection contains more than 150,000 boxes of prints, photographic films and associated records housed in satellite storage, and because much of it is very old there is the risk of deterioration. So we’ve now got the challenge of extracting the information from the analogue record, converting it into a high resolution digital format so we can put it onto our online systems to make it accessible.”

Supporting NCAP in its digitisation programme is Julien Sindt, Commercial Manager for EPCC, who has been looking at the most data-efficient way of digitally storing such a vast collection of imagery. The EPCC is a supercomputing centre based in the University of Edinburgh, with a reputation for leading-edge capability in all aspects of High Performance Computing.

Julien has helped NCAP optimise its robotic system to automate the digitisation of flat photographs on an array of 100 bed scanners, which can perform the task 24 hours a day.

Julien says: “We developed a computer programme that was able to reduce the time it took to crop a scanned photograph – the process of removing/adjusting the edges of an image to improve its framing – from two minutes per image to a couple of seconds per image, thereby minimising the amount of data that each scanned image takes by 40 per cent.

“It does this by cropping out all the extraneous information collected from the scanning bed, which is not the actual photograph, and this helps to reduce the image size. Typically, there’s a drop of 100 megabytes per image, which doesn’t sound like much until you realise that NCAP has 30 million images to store – that would be equivalent to around 7,500 terabytes. Using our programme, we can reduce this to 4,000 terabytes of data so effectively we are reducing the cost of storage of these digital assets by 40% per month.”

Decades-long process

Allan says NCAP is making progress but there is a long way to go: “We’ve currently got roughly 2.8 million images converted to an optimum high resolution format out of 30 million but our collection continues to grow as we’re receiving more material that is in the process of being declassified. However, it will be a finite amount eventually as digital aerial photography became established in the early 21st century, so I think it’s now realistic to say that we should be able to convert the whole collection in 15-20 years.”

In addition to providing online access to the collection, the digitised photography will also provide further value to researchers, particularly those interested in changes to land use.

Allan explains: “Due to the way the photographs were taken, you can overlap the images in order to stitch them two together to create a seamless mosaic that covers a large geographical area, and as there was a government programme to systematically resurvey the majority of the Commonwealth, from the 1940s up until 1980s, this can provide a snapshot of the regions which can have significant research implications, and especially for people looking into things like deforestation or climate change.”

Julien is also excited about integrating the data further: “We’ve got a couple of ongoing MSc projects looking at the digitised photography and one of them is trying to automate the ‘stitching’ process that can bring overlapping photographs together and improve the image contrast to make it easier for people to see the features of interest without losing any of the data from the underlying image.”

Immersive treasure trove

The other project is to enhance the 3D processing of these photographs, which are all taken at regular intervals from an aircraft with an overlap area of about 60 per cent between successive frame. Since each one is taken at a very slightly different angle as the aircraft travels forward there is the potential to take advantage of the stereoscopic effect to create a 3D image of the surface.

Julien says: “We’re testing whether it’s possible, from that slight angle difference, to automate the creation of 3D landscapes to estimate heights of buildings and other information. That is going to be a tough project as there are a lot of variables involved, but if we can, that would be fascinating.”

Allan adds: “So in addition to preserving one of the world’s largest collections of aerial photography we’re also producing a real treasure trove from a research point of view.”

_{Image credits: all – NCAP / ncap.org}

Tackling challenges for society through data innovation

Read more about Data-Driven Innovation