The world’s most complete digital twin of Alcatraz

Request a Demo

Capturing the biggest set of 3D scans of Alcatraz

The US island prison Alcatraz is a unique environment. The rugged 22.5-acre site of the notorious jail has now been digitally captured in unprecedented 3D detail, involving a world-leading project deploying multiple mapping technologies. Having camped in the jail’s infamous cells for three weeks during the cold San Francisco winter, project head Pete Kelsey and team member Dr Jeremy Sofonia tell us the backstory to this hugely significant endevor.

“This project would not have finished in the time we had been permitted…without the Hovermaps. No way, not a chance. They were a key component to the greater solution.”

Project Lead, Pete Kelsey, VCTO Labs

The Story Behind the Scans

Planning for Alcatraz

Pete Kelsey discusses the project objectives, why having a digital twin of the entire site was important to the Park Service and the many ways the data could be repurposed. He details the huge amount of pre-planning that was involved to ensure the project was a success – from getting permits to assessing which technology the team was going to deploy, to accurately and safely mapping every single space on the island in just 3 weeks – and why the Hovermap ST-X was the right tool for the job. Also hear about how the team overcame access challenges that included minimizing disruption to Alcatraz visitors, and how the project was managed on a day-to-day basis to ensure everything was going to plan.

Mapping the ‘Rock’

Jeremy Sofonia discusses the project origin and how it evolved from creating a virtual tour to developing a 3D model that would be used to monitor the site and plan maintenance and restoration works. Hear about the challenges of a typical day and how the team overcame the complexities of mapping the site, such as dangerous, dark, and difficult to access areas, by applying different scanning techniques. The sheer volume of data was immense and Jeremy details how the data was validated as it was captured. He talks about the workflows and structures that were put in place to process, manage and merge the data across multiple computers to ultimately deliver one accurate, detailed dataset of the entire site.

“The Hovermap is an incredibly useful tool for the AEC space. Why? Because of speed. If anyone would attempt to use terrestrial laser scanning to do what we did, they’d be there for six months. Probably more.”

Project Lead, Pete Kelsey, VCTO Labs

The Alcatraz tapes: interview transcripts

Emesent Hovermap and Aura played a key role in the Alcatraz digital twin mapping project. Team lead and project organizer Pete Kelsey, and Emesent’s Jeremy Sofonia, recount the challenges they faced and their key take outs from the historical 3D survey mission.

What was the Alcatraz digital twin mapping project?

PETE KELSEY: The Alcatraz project was for the National Park Service here in the United States. The objective at the outset, at the beginning of the discussion, was to create a baseline survey of the entire island.

All the structures inside and out that could be used as a baseline that all future surveys could be measured against. So, the genesis of the project was really about establishing a baseline survey that could be used for change detection going forward because Alcatraz – it’s a lot of things – but I think what most people don’t understand is that it’s a very dynamic environment.

The Park Service was keen to mention climate change, sea level rise, the effects of over a million visitors per year. It’s right in San Francisco on the San Andreas Fault, so it’s seismically active. There’s a lot of things going on that present challenges to what’s arguably the most notorious prison on earth.

So, they were very keen to get that baseline survey done. From there, the conversation changed into all sorts of interesting and creative ways that this kind of reality capture digital twin type data can be repurposed. That was the Genesis. 

JEREMY SOFONIA: The original idea of the project was to create this 3D model, a virtual tour, for the National Park Service to put content in a digital space online so that people could come and see and visit Alcatraz virtually.

And there are several benefits to that, notwithstanding the ability to get into places that are normally off limits. There are plenty of places, because of the state and the condition of the buildings, that it’s completely impossible to have tours moving through constantly. So this opens up more of Alcatraz to the greater public.

What sort of planning was undertaken for this project?

PETE KELSEY: The planning for the Alcatraz project…it took nine months. The conversation started in January of 2023. We began scanning in December 2023. What was involved in all of that? The first thing that comes to mind was just the permitting process, because Alcatraz is a national park that is visited every day of the week. It never closes. So, the visitation, the tourism, is ongoing all the time. The permitting process was for a scientific research permit. We had to craft that in such a way that it ticked a number of boxes for the National Park service, which were all relevant, in no particular order, to operations and maintenance.

I spoke about how dynamic the environment at Alcatraz is. Well, the Park Service has a well-documented and rather substantial backlog in maintenance throughout the park system, including on Alcatraz. Just the prison alone, parts of the other infrastructure, and buildings on Alcatraz – It’s almost like a layer cake. 

There’s Native American. There’s civil war. There’s the prison time itself from the thirties to the 60s and then from 1963 when it closed, until today. There’s all these layers of history and infrastructure that need to be maintained, because that’s part of the Park Services charter.

With regards to planning, there was the operations and maintenance piece, there was a biology piece – Alcatraz has always been a very popular spot for seabirds in terms of a rookery, nesting, and whatever. There is big concern about drones and the birds. The first thing: the survey had to wait until all of the young birds had fledged, and all the birds were off the nests. That’s why we did it in the winter, in December. 

There was this biologic component, there was the archaeological component – because there’s so much history – and there are biologists and botanists, and archaeologists and geologists, they all work for the Park Service. So, for my part of the pre-planning, I did my best to make the resulting data of the survey, as broadly appealing to the greater Park Service as I could, and the more that we talked about it with the Park Service, the more I knew it was going to be a great fit.

Yes, there’s a biological component to this, an archaeological component, operations and maintenance, certainly. 

The pre-planning was long. There were two permits – there was a scientific permit to actually do the work, and then the second one, and just as important, was the permit to fly a drone in a national park, because there are no drones allowed in national parks. Anywhere, no exceptions.

What was it like on Alcatraz during the project?

JEREMY SOFONIA: I don’t think anybody actually anticipated what it was actually going to be like. I certainly had no idea, but essentially a typical day was waking up at five o’clock in the morning. Leaving the cells that we were sleeping in and then going down to the administration offices where we set up our command base. We took over the National Parks office. We felt a little bit bad about that. But we had our computers and all the gear set up down there. So, by 5:15, 5:30, you’re down there. 

Staying on the island was camping style: sleeping bags, mosquito nets, headlamps, that sort of thing. Eating was the same. There was no cafeteria. There’s no food served on the island. So, we had to bring all of our food in. 

So, the morning was, you know, making coffee and heating up some instant oatmeal that you could make just by boiling water in the kettle. After a small brekkie, usually straight onto the computer to see what processes had finished overnight.

But there wasn’t a lot of time for that in the morning because we only had a couple of hours’ window before the tourists and the staff would turn up. So overnight, there’d be a National Parks guy that stayed with us. The rest of the workers and things like that would come on the island about half 8 in the morning.

So, we had a 2 to 3-hour window at most to capture data before crowds started coming. And that was really important because, sure, we can capture data with people, but that usually adds a considerable amount of time to our post processing to try to clean them out. But also, the National Parks, and therefore us, were very concerned to not impact on the tourist experience.

Tourists want to come into this place and reflect on the history, about what it’s like to be spending time in prison, and usually it’s a quiet atmosphere, even if there’s, you know, 1000 people in the room. It’s not an amusement park, it’s a very serious place. We were very conscious to not interrupt them.

So, if they were in reflection and thinking about being in solitary confinement, they don’t necessarily want us walking around with the Emesent Hovermap and disturbing them. So, it was a win-win to not be intermingling with them. That made it very challenging, though, because they’re there until 5-6pm, depending on the schedules of the ferries. That meant while they were there, we were focusing on the areas that were off limits. And so it was always this juggling act of what could we get done. There’s people coming, okay, where do we go next? And so it was difficult to plan day by day where we are going to go, and what we were going to do, because it was kind of constantly changing.

The good side was it’s such a large complex space that there was always something that needed to be scanned. So, we didn’t run into any downtime. If there were people there, we would go to this other place. If there weren’t people there, we did our scanning in this place. It was always about just trying to keep busy.

In the evenings, when we finished, and throughout the day, we’d try to keep processing data, just to keep that moving along. And then have a quick microwave-made meal, because again, it was that camping style. And, then it was nine o’clock. You’d be pretty exhausted and looking at going to bed. You’d be brushing your teeth in the public toilets. There are no showers that work on the Island anymore. So, you’d crawl back into your cell and then try to sleep for a few hours and then back up and repeat. On the weekends it was a bit better because our National Parks representative wanted to spend time with his family, which is more than fair on the weekends. So he went home on the weekends and that allowed Pete and I, and some of the other guys, to go back (to San Francisco) and get hotel rooms and that gave us a shower for the first time in the week. It all worked out in the end, but it was definitely not anything that we could have foreseen planned.

How was Emesent Hovermap used in the Alcatraz project?

PETE KELSEY: So once those 2 permits were issued, things actually happened pretty quickly. I knew that there was going to be an airborne component and a terrestrial component, an interior component and an exterior component. The airborne, obviously, it was all drone-based, all LiDAR-based for one particular flavor of LiDAR or another: we had GNSS, we had SLAM. We even had terrestrial laser scanning on the ground for the terrestrial component. We had photogrammetry and we flew multispectral because some of the biologists were interested in the plant life, which is on the island as well. 

The biggest task was the interior spaces because there’s the famous prison, the cell house  – of course widely known around the world – but there’s a hospital, kitchen, cafeteria, and morgue. There’s the powerhouse that generated electricity for the entire island, all that infrastructure, and a lot of it is not accessible to visitors, much less anyone, because it’s in bad shape or it presents risk to life and limb. 

But we had access to it all. I knew right away that the only way to do all these interior spaces in the time we had was going to be with a Hovermap. Just because it’s mobile and can be deployed in a number of different ways.

We were exceptionally lucky because we had two Hovermaps on site for that portion. And without the Hovermaps this project wouldn’t have been successful. We wouldn’t have got it done. We basically had three weeks to do every single space on the island, interior and exterior, and we actually used the Hovermap for both, but mostly for the interior. 

Getting into the workflow a little bit, we had to have a fair amount of the exterior done as well. So we could use that data to match with the airborne drone-based LiDAR. So the Hovermap, both of them fortunately were the newest model the ST-X – with its fabulous range, was the right tool for the right job. This project would not have finished in the time we had been permitted for without the Hovermaps. No way, not a chance. They were a key component to the greater solution. 

JEREMY SOFONIA: The primary objective was to do handheld mapping of the interiors. That was Emesent Hovermap’s role in this case. So just your basic handheld mapping was used most of the time. But because many of the spaces are not well lit –  or even lit at all – we had a special lighting rig that the engineering team had put together that snapped right onto my Hovermap unit and could illuminate the field of view of the GoPro. The GoPro was was there to get RGB true color data. 

Not only were we looking at the LiDAR data, but colorized so that blue rubbish bins are blue and the rust on the wall looks like rust. Adding that element of color to the requirements increased the challenge and the difficulty significantly. One of our issues was, how do you light these large spaces or these small tunnels? This lighting rig that I had on the handheld units worked great. It did its job perfectly.

For those hard-to-reach places I also brought an extension pole and used it a few times. A good example of this was on the top of the lighthouse. I knew that the top of the lighthouse was going to be seen by the drone data, but I wanted a complete Hovermap model for our own purposes as well. So I was able to walk the interior of the lighthouse and we were able to scan the exterior easily enough, but I couldn’t get up to the tip-top of the lighthouse without climbing another small ladder. That just looked like a bad idea. Plus, the lighthouse is an active Coast Guard station. Going on top of that would have required seeking additional approvals. It just wouldn’t have been worthwhile. We didn’t want to bother the Coast Guard with something like that. So, by being able to put the Hovermap on a pole, I could do a walk around the top, along the balustrade of the balcony of the lighthouse. It saved me from having to climb up and accessing a part that would have been potentially dangerous, but also requiring additional levels of approval. 

Lastly, Boston Dynamics. They arrived with Spot, the robotic quadruped dog, and we used that in a building called Building 64. It’s the old accommodation units for the guards and their families. So, think about a massive apartment complex. People don’t realize this, or at least I didn’t, that the guards didn’t live in San Francisco and commute back and forth. Their entire families lived on the island. They had a school. They had everything that they needed, more or less. So, this old housing building, multi-story, large complex space is actually closed with signs on the door that say: Danger Lead, Danger Asbestos. And for those signed levels, we were able to use Hovermap on Spot to go in and capture that data because Spot is not affected by lead or asbestos the way that we would be.

It would have taken us longer to put the PPE on and take it off than it did for us to just do the whole job. It’s a great example of how we can use robotic systems to not just keep ourselves safe, but to minimize the red tape and the time consumption that it does take, not just for us, but even for the health and safety people that have to review these things. It’s a massive time saver all around. 

What were some of the Alcatraz project’s challenges?

JEREMY SOFONIA: More than a physical, it was mostly a mental challenge. I need to walk on this floor, but is it safe to do so? I need to climb this ladder, but is it safe to do so? Before we would do anything like that, we would speak with our National Parks representative and have surety that that was a safe thing to go and do. That didn’t make it any more fun or less intimidating, but we were always able to validate or show that something was safe before we would fully jump into it. 

I think that’s an important element to understand. We weren’t taking silly risks. Although some of the things and some of the places felt very unnerving – just because of the environments and the age of the assets.

PETE KELSEY: Access is absolutely number one. And what I mean by that is with every room there’s a door. With every door there’s a lock. Do we have the key? If we don’t. Who has the key? Are they working today? Do we know where they are on the island so we can go find them and borrow the key and then return the key?

It was the first few days that we knew right away that that was a problem. And the National Park service we’re just great. I asked the facilities manager there on Alcatraz. I explained to her the problem we were having and she didn’t blink. She got up, went into a safe and handed me the keys to Alcatraz.

And I’m not even exaggerating, the master key that worked in every door and every lock for the entire island and everything changed. Then, I understood we’re going to get this done in the time we have. That risk went away. So access was by far number one. 

Not disrupting visitors – huge. It was very important to the National Parks service, very important to us, because people have come from all over the world and spent thousands of dollars to come and have this experience at Alcatraz, and the last thing we wanted to do was spoil that in some way.

So, working around them as best we could was a challenge, but I think we did really well at that one as well. 

The weather, for all the exterior work. This is San Francisco Bay in December. It was cold, wet, completely socked in with fog more often than not. But on the two days we were allowed to fly: perfect weather, bright sunshine, no clouds. It was a bit of a miracle. 

On a personal note, we were the first group, the entire team, to spend an extended amount of time living on the island since the Native American occupation in the late 60s, early 70s. Meaning, we lived on Alcatraz, sleeping in cells for three weeks. And no one had done that in over 50 years. What did that mean from a challenge point of view? We had to bring in all our own food. Because there’s no food allowed on Alcatraz. Anywhere. They don’t sell it. You can’t bring it on the island. So we had to bring in all our own food. So it was a lot like camping.

Well, what food do you bring? You can’t bring anything that’s going to spoil. So it’s all freeze dried, just add water kind of thing. 

Sleeping, I’ll never forget the first couple of nights sleeping in D block in cells in Alcatraz. It’s all concrete and steel. I mean, you hear everything. And a couple of my team members snored like, like I’ve never heard. And so none of us, none of us got any sleep the first couple of nights, but we moved some people around and got over that. 

The first night, big surprise. We got eaten alive by mosquitoes and that was a complete surprise. Well, the next day I went into San Francisco and bought every mosquito net in the city and hung them over our bunks and we solved that problem.

But the biggest one, the one sort of go, no go for this project was being handed the keys to Alcatraz. Without that, it wouldn’t have worked. No way. 

How did you go about organizing the huge volumes of data captured on Alcatraz?

PETE KELSEY: First and foremost, I put together an amazing team and included friends of mine who are well versed in the IT back-office file management component of this. Without a scan plan, without a naming convention, without some form of standardization, this wouldn’t have worked. It would have been absolute chaos. 

So how did we do it? Fortunately, a lot of the big pieces call them a single scan. There was a single airborne GNSS LiDAR flight. There was a single multi-spectral flight. There was a single photogrammetric flight. Where it got challenging and where we really had to stay focused was with the Hovermaps.

Because at last count, I still can’t get my head around that, we are at over 200 Hovermap scans, which I’ll bet that’s a first. So, to be able to know what each one is called…it started with where are we? So, for example, CH for cell house, and then what room or what block of the cell house and that sort of thing.

And I’ll be honest, we spent the first couple of days really, really overthinking if it’s possible. It’s what happens when you have a lot of smart, very capable, very talented people in a room and we all come with our habits and “we should do this, we should do this”, and we’re all saying that’s such a great idea, and then I’m in the background saying: “Guys, this is all really important, but we have to start, we have to get going. The clock is ticking.” 

So, essentially, I knew that Jeremy Sofonia from Emesent was going to be the main guy. The Hovermap work was going to fall to Jeremy and myself with Jeremy being way out front in terms of the workload because I was project managing, and all of that. So, it basically fell to Jeremy to create a system that would make sense to all of us, but most importantly what worked for him. He did it and we’ve had no issues with that today. It’s been fantastic.

Were you doing any data processing on Alcatraz Island?

JEREMEY SOFONIA: Absolutely. For two reasons. First, just the sheer volume of data. It’s great to try to keep up with it so that you don’t have this massive backlog of data to do, but also it’s a quality assurance, quality control issue…QA, QC type of approach where we wanted to make sure we got what we came for, as we were collecting it, because of the time and effort it takes to get there. If we missed something, or if a data set didn’t happen to turn out the way we hoped, going back is not easy, particularly for me from Brisbane Australia.

It’s much more efficient to just check that what you’ve got is true and correct and then carry on. We captured easily over 100 scans throughout the whole project. I think in the cell house alone I have 28 individual scans. That sounds like a lot, but it touches back onto that flexibility that we needed to have. I might be in there scanning tier 2 of B block. And then I realized that there’s a crowd coming in. Okay. Well, I guess I’m not going to do tier 3. So, you’d have to stop. And so breaking up the cell house allowed us to be able to capture data in there where we could, and then go do something else and then come back.

The idea is that we could process that data as we went, and out of all the scans, maybe two or three had a small slip. So you had to make a decision as to whether or not it was faster to try to reprocess it with different settings or just walk over 10 feet and scan it again. It allowed us to do that QA QC. 

Pretty much all processing was done in Emesent’s Aura software. With that initial processing, you’re able to use the visualization tool, which can load the point cloud very quickly, and essentially what we’re looking for is did the point cloud slip. 

The SLAM algorithm that we use is extremely robust, but it can often be just user error. Maybe I got too close to a wall. If I was in a small space, maybe I blocked Hovermap’s field of view, and it wasn’t able to link up this room to the previous room. That sort of thing. It only happened maybe twice during the whole project, but by visualizing the point cloud by time, it’s a very quick and easy way to see if the point cloud has slipped, and you basically do that by looking to see if you close the loop, that you start the scan where you’ve finished the scan. Where you started you should see the same object. In different colors, if it’s colored by time, they should be on top of each other and there shouldn’t be two fire extinguishers where there’s one, for example. 

So, you can see if there was a problem very quickly that way. And that allowed us to validate that scan and move on. 

What did the Alcatraz project teach you about Emesent’s technology?

PETE KELSEY: I have been working with Hovermap and Emesent for four years now, maybe five. My background is AEC – architecture, engineering, construction – and I knew from the first time I saw the Hovermap I knew the potential for it to be an incredibly valuable tool in the AEC space. That’s why I leaned in and started collaborating with Emesent.

I just knew that it could do that. At the time, the focus of the Hovermap was in GPS-denied areas, specifically mining, which it is the the perfect solution for any GPS-denied environment. So fast forward to Alcatraz and Alcatraz is a classic AEC project. This is architecture and engineering there. I don’t remember the count of actual buildings we did with the Hovermaps we had, but it’s easily 40. To cut to the chase, my hunch was right. The Hovermap is an incredibly useful tool for the AEC space. Why? Because of speed. So, if anyone would attempt to use terrestrial laser scanning to do what we did, they’d be there for six months. Probably more. 

Now okay, terrestrial laser scanning might give you a higher level of detail, but that wasn’t required. As I said before, the Hovermap was the perfect tool based on the time it took in the field. So with the available field time, the Hovermap was the only way to do it. 

In terms of things I learned about the Hovermap, there were a few, a few nuances more specific to SLAM rather than the Hovermap itself in terms of just good field data capture do’s and don’ts to try to avoid – the dreaded slips and drifts and what not, that’s inherent to all SLAM technology. I learned a few things from watching and scanning with Jeremy, in terms of data capture, but I think most of what I was not aware of is just how much Emesent’s Aura can do, and specifically for a project like this, because each individual scan has value unto itself.

But to be able to merge over 200 scans together, which I seriously doubt anybody has tried before, that to me is where I get to sit back and just say, AEC, this is such a great fit. We just did a 27-acre island with over 40 buildings inside and out with all of the interior spaces, almost without exception, done with a Hovermap. It’s great.

I knew that the workflow existed, but I had not yet tried geo-referencing Hovermap data with say airborne GNSS LiDAR data. Never done that before. Super cool, because that ticks a lot of boxes in the AEC space as well. Especially with my history on the civil survey geospatial side, if it’s not geo-referenced and it’s not survey grade, a lot of the people who I have relationships with would just say pass. I can’t use it. Well, I can now look all those people in the eye and say the Hovermap can do that. 

We can geo-reference. We can get survey-grade data accuracy. We can integrate with any other type of LiDAR sensor. You know, a point cloud is a point cloud is a point cloud. And all the Hovermap is doing is creating point clouds faster than just about any other sensor.

What did the Alcatraz project teach you about LiDAR scanning?


I’m not sure that many – if any – projects that we’ve ever done at Emesent, or even our customers have done, have been this complex in this short timeframe. So how do we handle this volume of data? The fact that it was in color. Some great lessons there.

I probably captured more colored data sets in the last three weeks at Alcatraz than I have in my entire life combined before that. So thinking about the techniques on how we scan with color. So we talked about lighting a minute ago, but also how do I enter the room? How do I cross the room? Is it good to walk down the middle of the room, or do I hug the walls? Do I go to the corners? Do I do a circle? Just really thinking this through you’re able to experiment a little bit on the fly to try to identify what’s going to be the fastest, most efficient way to cover this without missing anything. And so there was a great lesson learned for me about how I enter spaces with color and I end up doing this kind of zigzag pattern from corner to corner to corner.

And it doesn’t matter if it’s a large room or a closet. I still get in the closet and I try to go from corner to corner to corner because you want to make sure that you are seeing every angle possible, because you need that color information and you need points to have that colored point cloud.

Book a Free Demo