Back To Debugging

This afternoon Jeff, Wentao, Zoe, and I went out for another boat deployment to test our image detection to path planning pipeline. Unfortunately we continued to experience diving problems, and we believe we’ve narrowed the problem to the Iver being unable to track the bottom of the sea at deep depths (>80m).

Jeff, Sam, and Wentao calibrating the compass.

 

Zoe guiding the robot.

When we came back to shore, Amy and Sam helped us carry the robot back to the apartment, while Roslyn made everyone delicious dinner. Thanks again Roslyn!

Tonight is the Isle of MTV concert, so Amy, Roslyn, Sam, and Wentao went to experience the largest free concert in Europe! Photos to be posted… Jeff and I stayed behind to continue debugging. Props to Jeff for sending many emails and making phone calls to get more information about how to fix our problem.

Debugging is always the hardest part of robotics, but that’s also what makes it so rewarding when everything works. We will be up early tomorrow for another deployment, and we hope to get Vernon back up and mapping wrecks soon.

Iver Meets Diver

Today Jane, Amy, Zoe, and I went to Elmo's Bay to do another deployment. We were wanting to check that Vernon was still diving ok, as well at try to get good data of the shipwreck there. We did one pass over the shipwreck, 5m above the bottom. All went well, but the video data we collected was very murky, due to a lot of divers around the wreck kicking up sediment. We then ran a second mission going at 3m from the bottom. After the second dive Vernon didn't come back up for about 15 minutes, and we were concerned since each line was only supposed to take one minute. Zoe had gone and talked to some divers to see if they could spot the AUV and bring it up for us, but just as she did we saw Vernon emerge on the surface. Much thanks to the diver who helped release our robot after it got caught in the shipwreck! You can see on the GoPro footage above the AUV getting stuck and then being released by a diver.

Jane at the Deployment in Elmo's Bay

After coming back from deployment, I have been working on some more realistic graphical representations to go with our underwater meshes and make them actually look underwater. Today I finished making my first skydome, to make the beaufighter wreck look like it was underwater.

Skydome around the Beaufighter

Though we got stuck in the shipwreck on our second mission, we got a lot of great GoPro footage since we were so close. So I have also been working on collecting the frames from the video data, and seeing what 3D meshes I can get from them.

Our Day as Gozitans

Sunday, funday! Today, I had the pleasure of waking up early to go to water polo practice in Sliema. I have had the absolute best time playing with these ladies and appreciate the friendships I have been able to build through our common love of the game. It is amazing to me how after only a week in Malta (and help from Timmy), I was able to play the sport I love and have friendships that will hopefully last much longer than the short week we have left here.

The polo girls at practice at the Sliema Pitch.

The rest of the research team came by the pool to pick me up and we made our way to Gozo. Our first stop in Gozo was the Ġgantija temples. We were amazed at how tall they were and had a great time running around with such an amazing view of Gozo. The most interesting part of the temples was how art was used to keep a record of what they looked like before they were destroyed and their history over the last seven/eight centuries.

Sam, me, and Jeff at the temples.

The view of Gozo.

Once we finished being good archeologists, it was time for some well deserved rest and relaxation. Most of the beaches in Malta are either made of rock or covered in them, so we set out to find a sandy beach to better fit our Californian needs. The beach we went to was crowded and still a layer of rocks right when you got to the water line, but it was beautiful and just what we needed! Jane even found a good use for the rocks at the beach.

Just a girl and her rock.

Beach day!

After some very exhausting time at the beach (spent mostly floating in the sea and sleeping in the sand), we went for dinner in Xewkija where the Feast of St. John was happening. We had great food in a gorgeous place, which seems to be a common theme for this trip.

Finally, we all got some rest on the ferry and bus rides home to Sliema.

Super Saturday

This morning we walked over to the University of Malta to have a presentation on using Agisoft Photoscan. This is the software we use to create 3D models and textures of objects from images. The presentation was given by John Wood, one of Timmy's good friends and dive buddies. He gave us some tips on how to get pictures that would work best for Photoscan. We also learned more about diving and got to see some pictures of John, Timmy, and the rest of their team diving some underwater wrecks. Very cool!

Agisoft Photoscan presentation by John

Cooking up a storm for lunch

After the presentation we stopped at an Asian food store and got ingredients for lunch. Jane, Roslyn, and Wentao made some fantastic food! For the rest of the day we were free to do whatever we wanted. Roslyn and Amy went to the beach. Jane and Wentao stayed at the apartment to relax. Jeff and I went to the store to get some baking ingredients after working on our georeferencing script a bit. Apparently molasses is called black treacle here!

The team with John

Amy with some delicious ginger cookies (and Zoe being a goofball)

Today was a great chance to relax and enjoy some time together. Tomorrow we're planning on taking a trip to Gozo and then it'll be back to work for our last week!

Vernon Cleans Up the Sea

Today the team spent the morning working together in the apartment. In the afternoon, Roslyn, Wentao, Zoe, and I went out on another boat deployment, while Amy, Sam, and Jeff continued to work.

More laptops = more productivity.

The plan for the deployment was to test the full pipeline from Jeff and Wentao's image detection algorithm to my path planner. Though Jeff and Wentao are still working on converting the raw sonar scans to position-corrected geotiff files, they were able to give me the coordinates of a few sites of interest. With this information, I generated cross-shaped lawnmower patterns for two sites with my planner.

About half an hour into the first mission, we quickly realized that every deployment always brings a new surprise. Wentao noticed that Vernon had aborted its mission in the middle of visiting the first site, so we drove to where it was parked and checked for problems. Sure enough, there was a long piece of plastic that was tangled in Vernon's propellor. Today we learned that Vernon can not only map shipwrecks, but also clean up trash in the ocean!

The gnarly piece of plastic stuck in Vernon's propellor.

Following this mishap the motor appeared to still be functional, so we decided to continue by visiting the second site. However, GoPro footage and sonar data suggest that Vernon had trouble diving during this second mission, so we plan to revisit both sites next week after debugging the AUV.

Wentao action shot.

Roslyn guiding Vernon in the water.

Meanwhile, Amy, Sam, and Jeff spent the afternoon at the apartment, where they could finally work without me "being so loud." Amy and Sam extended work on camera position planning for photogrammetric reconstruction, while Jeff continued to work on the shipwreck detection algorithm and sonar file conversion.

In other updates, Roslyn has been working on displaying a JU 88 plane wreck reconstruction on her website, as well as adding a feature that displays wreck locations on Google Maps.

After the deployment, the team went to a Sliema vs. Sirens water polo game at the National Pool, where we reunited with Chris and Timmy.

A close game.

Finally, we ended the day with a wonderful dinner with Timmy at Surfside. 

More from June 22nd

This video clip describes our amazing collaborator, Timmy Gambin's, work, (which we are a small part of this excellent research initiative):

History Beneath the Blue

Here are a few more images from our productive day yesterday

The boat!

Sam putting the fin on and Jeff preparing the mision

Amy helping get Vernon in and out of the water

Happy about a productive afternoon

Working hard processing data

making serious faces (on purpose)

Unexpected Success

Today we planned to do an afternoon deployment testing a new strategy for scanning. Last night Wentao and I thought we had made good progress on the new scanning strategy so it could be easily finished this morning. However, Wentao, Jane, and I ended up working right up to the time we had to leave getting the missions ready for today. There were moments where we considered not going out today, but when the time to leave came around we had enough ready we figured it would be worth it.

Everyone helped carry the equipment done to the boat launch. Sam, Amy, Zoe and I went on the boat while Wentao, Jane, and Roslyn stayed back at the apartment to work on research. The first mission launched well, but as we neared the first surface point Amy spotted Vernon surfacing right under our boat. We were able to accelerate over Vernon before he surfaced so there was no collision, but it was a little scary. We had another small scare when fetching Vernon from the water where the water pushed Amy close to the boat's propeller. Other than that the missions went well, despite being rushed to prepare them.

Sam and I hugging after the successful missions

Another deployment and Underwater Object Detection

Yesterday, Chris, Jeff and I took Vernon on a mission to survey a brand new area.

Vernon on the industrial boat

In the afternoon, we continued working on our own research. Jeff and I are working together on the identification of potential heritage sites (shipwrecks, planes etc.) from side scan sonar images. Our method consists of two stages, object proposal and classification. Our object proposal pipeline uses rectangular filters to look for shadows of objects in the sonar scans. It works quite well on our test data set and is able to capture all the targets.

However, the purpose of object proposal is to filter out most uninteresting areas in the scans in a quick and simple fashion, so it produces a lot of false detections. We are working on the following classification stage that uses machine learning techniques to reduce false alarm rate and make our detections more precise.

The rest of the team had a productive day working as well.  Roslyn is working on getting some new models ready for the web, Sam and Amy worked on getting openCV up and running with their openGL application and Jane worked on literature review of PRM and implemented an RRT in python.

Animation, Aviation, and America's Embassy

Today was a very full, adventurous day! To start, Sam, Jane, Chris, Zoë, and I went out for an early morning deployment at St. Elmo Bay. We already have sonar of the wreck at this location, but we wanted to get more data (including bathymetry and video) for the visual reconstruction that Sam and I are working on. For an update on our work so far, we have taken Katie Davis' PRM camera code and put it into new code from what previous ICEX teams have been using. In this new code Sam and I have modeled/animated current, surge and swell of particles, bubbles, seaweed, and have an imported 3D model of the shipwreck. We are working on adding god rays and caustics to more accurately light the scene. We would like to use the video data to have a better idea of how much light should be in the scene, what types/how many plants there should be, and in general to make the scene more realistic.

Screenshot of Sam and my work so far.

After leaving Valletta, we went to lunch at the US Embassy. Zoë has written about our experience there -

"We had the opportunity to have lunch at the U.S. embassy and chat with the ambassador and other diplomatic staff about our research.  They were excited to hear about our work and shared some stories about their lives in the diplomatic corps.  The ambassador spoke about their joint work with Timmy to designate certain maritime World War 2 wrecks as war memorials and the importance of these sites in terms of history and honoring the historical connection between Malta and U.S.  We enjoyed meeting the embassy cat, tinker bell, and strolling through the grounds to see their solar array and wind turbine.  The staff joked that next time we should plan to conduct robotics outreach with them as they love to learn new things - maybe we should plan that for next year.  Hearing from the staff about their various placements at other diplomatic missions was a great reminder of the scope and importance of building relationships around the world."

All dressed-up for the Embassy!

Then it was time to work, so we headed back to the apartment. People coded, did research, and got in some well deserved exercise time. At six, Anthony Burgess came by to teach us all about "Aviation Archaeology Underwater," and it was really interesting. I was sad at first because I had to miss water polo practice, but he gave such a good talk that it was worth staying! He even told us he can send us a drawing of the plans of the beaufighter that we have collected sonar data for, which is great because we can always use more data!

After the lecture on aviation archaeology underwater with Mr. Burgess.

We finished the day with dinner and a walk on the waterfront. Roslyn was happy because there were plenty of cats that needed love.

Roslyn wants to keep all of the cats.

(For any readers that do not know, I have been practicing water polo with the Sliema and Women's National teams since arriving in Malta. I have Timmy to thank for setting that up for me. I love everyone I have met through water polo here already, and I am so grateful for the unique international experience that I have so far!)

Beaufighter Wreck Footage!

Today Roslyn, Chris, and I went out for another boat deployment! This time we finished surveying the area Timmy asked us to scan, and we also visited the Bristol Beaufighter wreck off the coast of Sliema.

For the Beaufighter wreck, we used a star-like path that was generated by my path planner. Specifically, the AUV completed 4 lawnmower scans, each rotated by a 45 degree angle to capture a variety of viewpoints.

Screenshot of the Beaufighter mission plan.

The goal of this mission was to gather GoPro footage of the wreck for photogrammetric 3D reconstruction. All in all, the mission was a success, and this was the first time we got quality video data of a wreck. We still have 2 hours of video to fully process, but here is a quick glimpse of what we got. Enjoy!

Displaying Meshes online

I've been working on displaying some 3D models online of the shipwrecks we find. So far I have been using two meshes from last summer, and I created a mesh using AgiSoft from photos taken last year of the Beaufighter plane. I am hoping soon to have more image data of the shipwrecks in Elmo's Bay and off of Manuel Island. Once I have imagery I can create more 3D meshes to display. I have developed a website and the goal is to educate people on many of the underwater treasures around Malta. The 3D models are interactive, so users can look all around them and zoom in and out to get up close visions of the shipwrecks. Below are some screen shots of the meshes I have been working with so far. I had to do some modifying of each mesh in Blender to get the textures to work, center the mesh, resize it, and delete excess pieces of the model that were unnecessary or inaccurate.

Here are some examples of other websites I have been basing some of my work off of:
https://sketchfab.com/models/9ee29137e6ff40c1b70388b5cffa13fe
http://www.seriousanimation.com/

Another Day at Sea (and no WiFi)

Yesterday we had the pleasure of not having to deploy until early afternoon. This was a great time for all of us to work on our programming for a while and to create some helpful scripts for dealing with the AUV. We also figured out a more convenient way to transfer data between the AUV and our computers via USB and Samba sharing. Unfortunately, our internet decided to take a break which made communicating and working a bit more difficult.

Vernon going out on a mission

At 1pm, Chris, Jeff, Roslyn, and I headed out on the boat to get a few more lines of sonar data. The water was very nice and calm. The AUV's mission went smoothly, until the last stretch where we didn't spot the AUV surfacing. After searching for a while, we decided to head over to the final location of the AUV and crossed our fingers we'd find it. At around minute 75 of not seeing the AUV we spotted it at its final destination. Phew! With about 15% battery left on the AUV, we also decided to do one short test of the AUV's bathymetry capabilities.

Hoisting Vernon out after a successful mission

After getting back to the apartment we took a look at the data and one object detected looks very promising! We will most likely go back and further investigate the possible shipwreck. After that bit of excitement, we all decided to wind down and play some games. Hopefully the WiFi will come back soon!

Exhausted Vernon and Maltese Corsairs

As a penalty of its little misbehavior yesterday, we decided to make Vernon work really hard today. The plan was to do two trips to the survey area, one early in the morning and one in the afternoon and with a charging session in between.

Morning team heading for the boat

The morning team left at 6:30 am. Amy and Jane made two missions to expand our surveyed area further from shore. It was a calm day at sea, and the first mission went so smoothly that the team found some time to code, read, write and draw on the boat.

Fascinating view of Valleta from the boat

Karly's masterpiece

At the end of the first mission, Vernon had already done five passes over the survey area, which is about 3km long, and its battery had dropped to 21% (running sonar takes a lot of power), but we decided to push its limit and do a sixth pass. This time, we included a safety return path in the mission, so that Vernon would drive to the parking location in case the battery dropped too low. However, we waited at the parking location and Vernon did not show up at the appointed time. We had to backtrack again. Fortunately, it only took several minutes before Amy found Vernon, exhausting all its power and floating helplessly on the sea.

We agreed that we should not do more than five passes over the area in future expeditions. The afternoon mission was canceled because Vernon was so exhausted that it could not be fully charged in time.

In the afternoon, we had an amazing presentation on Maltese history by our guest speaker Liam Gauci. We learned about how the Knights of St Johns fended off the great siege by the Ottomans in 1565 and built the city of Valletta, named after Grand Master La Vallette, the hero of the siege; how corsairs, "legal" pirates supported by the Spanish court, flourished the economy of Malta by raiding Ottoman merchant ships and selling their cargo to Europe; and how Maltese culture is influenced by cultures from all over the Mediterranean world such as French, Italian and Arabic. For instance, the world for God in Maltese is "Alla", which is similar to the Arabic God Allah. The team also got really interested in Maltese cuisine after the presentation. Stewed rabbit has made its way into Roslyn's bucket list.