Developer Blog - Improving Sailaway's Ocean Height Map Accuracy
Hello Pioneers,
This is Richard Knol, the developer of Sailaway. I’ve been behind the scenes more recently as I have been busy working on some big changes to Sailaway. The biggest of these tasks has undoubtedly been improving the accuracy of the map data so I thought I’d give you an insight into this process in my first developer blog!
Sailaway’s Map Data
The Sailaway maps are all rendered in real time based on the 3D model of the world that is stored on the Sailaway servers. This model was made with data from NASA and a few other sources and it holds the height of the terrain or sea floor at any location in the world.
There are a few issues with that:
The NASA data only provides the height of the land, not of the sea floor. It also contained measurement errors and it has a resolution of about 10m2, 30m2 or 240m2, depending on the area.
Most of the underwater height maps I found only have a resolution of about 240 meters
The underwater data I found and could use is inaccurate and incomplete with regard to coastal waters
Not all data sources use the same corrections for the fact that the earth is not a perfect sphere
So all in all a rather depressing list of issues when you want to run a worldwide sailing simulator.
Luckily there are nautical charts available that give more accurate info about water depths and much more. But charts are 2D and Sailaway uses a 3D model. Basically, a chart tells you that there are regions colored light-blue where the water depth is 0 - 10 meters, and regions in another shade of blue where the water depth is 10 - 50 meters. But what happens in between?
Following that data strictly would turn the entire seafloor into a giant staircase. Additionally there are soundings on those maps that give the exact water depth at a certain location. But they don't say anything about if that point is a peak with a steep slope or a single measurement on a horizontal and flat surface. So here's what I did to improve the Sailaway 3D model with the US Electronic Charts as provided by NOAA...
Deciphering the data
First the charts were downloaded. All 1280 of them. Next they needed to be read, or rather deciphered. The format in which they come is called S-57 and it was defined by the International Hydrographic Organization decades ago. It dates from the days when computer scientists tried to write data dictionaries and press various record types into a highly efficient binary format. A time when JSON and XML were never heard of yet! So I deciphered the gibberish and extracted the data that I needed for Sailaway and pushed that into a JSON format. This was several weeks work, but in the end I had a neatly organized collection of 1280 JSON files and
1280 small overview images of the actual charts.
The US charts come in 6 levels. At level 1 there are large overview charts, building down to level 6 where a specific port is shown in high detail. I wrote a program that would process each of these maps starting at level1. It would download a section of the Sailaway 3D model, match the Sailaway depths/heights with the data of the chart and make the necessary corrections.
Unfortunately that technique didn't work, because it would create a staircase seafloor after the depth regions of the chart are applied. And it would litter the seafloor with spikes or pits at the locations of the soundings in the chart.
So instead I used the charts to verify our own data and make gradual adjustments where needed. If the chart says the seafloor depth is between 20 and 200 meters, and Sailaway has 20.5 or 198, I leave it untouched. If the Sailaway model says 18 meters, the floor is lowered and also the pixels around it are lowered gradually to smooth the sea floor. This way I kept as much of the shape and structure of the seafloor intact as I could.
Then there was the problem of turning land into water or water into land when the coastline would not match the chart. Turning water into land without any information on what this land would look like will result in ugly terrain. There is no fix for this. Sorry.
Implementing the data
So now that the algorithm seemed to work (again a few weeks work) I could put the computer to work. Starting with Level 1, here is how the numbers look:
The Sailaway model is made up of 0.25 x 0.25 degree tiles. That means (360 x 4) x (180 x 4) = 1,036,800 tiles. The tiles have a resolution of 900 x 900 pixels. Each pixel is represented by 2 bytes. That means 900 x 900 x 2 = 1,620,000 bytes per tile. Times 1 million tiles is 1.5 TB of data, or 800 billion terrain heights.
To process 1 chart of Level 1 (typically a region of 20 x 30 degrees) took about 2 days to compute.
In the meantime, I read and imported all the buoys, lights, rocks, shipwrecks and other vital information from those charts into the Sailaway database. This resulted in 120,000 new objects all along the US coast.
So my computer was humming and after many days, when it was already processing the charts for Level 3, I discovered a stupid mistake. Imagine a tiny volcanic island somewhere. A beautiful thing in the Sailaway maps, where you can see the slopes and the crater clearly. But this tiny island is not big enough to have made it on the Level 1 overview charts. Instead it falls in the big area indicated as 200-2000 meters deep. After processing the chart, the entire Island has vanished and has been turned into 200 meter deep water. Of course when the charts of level 3 are processed the island is reinstated, but all of it's characteristic volcanic shape is gone. It is now a flat surface a few meters above sea level.
As a result, the algorithm had to keep a copy of the original shape at hand. And every time it wanted to alter the current height of the land or sea floor it had to refer back to the old data and see if it could use any of the original height info to recreate some of the original shape of the terrain.
The algorithm was started again from scratch. And after many weeks it is now ready with correct calculations. Yay!! The result is a more accurate seafloor, and a very ugly coast. The last is because after processing the 2D charts, a pixel is either land or water. Imagine a nice coast with some rocks, some distant mountains, small pebble beaches between the rocks. The chart may have drawn a line over that coast, dividing it into land on one side and water of at least 20 meters deep on the other. The result is a coast of steep cliffs, that plunge into 20 meter deep water. The pebble beaches are gone, and because of the size of the pixels the coast looks like a horrible collection of zigzag cliffs.
To fix this, or at least make the coast look somewhat more realistic, is my challenge for the coming weeks. Wish me luck!
Ongoing map improvements
I started work on these map improvements early August. It is now the end of October and I am not done yet and this was only the US coast! The European and Australian coast will be even more challenging. But it is great fun to work on this and see the maps improve. Sailaway becomes more mature with every step we take.
I see players asking for more detail in the terrain often; for ports & marinas, and even trees and buildings. These would be awesome to have, I fully agree. But when you realize that the current work in progress coastline, made up of 10 x 10 meter pixels, already measures 1.5 TB of data. Just imagine the amount of data required when this includes the rest of the world! All that data has to be collected, processed, checked, maintained, refined and find its way to your computer somehow… don’t forget, Sailaway is a persistent online world.
The world is a really big place and I hope that the process behind how I implement this accurate data has become much clearer. Please do not expect the same level of terrain detail in Sailaway that you see in other games. They do not have to carry the weight of the world with them.
Thank you to all of you who have joined us in early access and support this project! I’m sure you can see from the process above, this is a passion project for me and I’m thrilled to see so many people sailing the seas I’ve created.
- Richard Knol, creator of Sailaway.
Weekend Races - Three more races to keep you sailing this weekend!
Hello Pioneers,
Here are this weekend's races!
Remember, if you don’t want to sail alone, you can look for a crew in the Multiplayer Forum, there are plenty of sailors looking for crew members or boats to sail with. Perhaps some of you more experienced sailors could team up with newbies and spread the knowledge?
Long Race
São Luís Long Race
Saturday 28th October - 19:00 BST / 20:00 CEST / 11:00 PDT / (Sunday 29th) 03:00 JST
Est. Race Distance: 115NM
Est. Race Time: 23 hours
Weather report:☀️ More detail A race from the picturesque city of São Luís - the race starts with a long upwind stint in a narrow channel so keep an eye on your waypoints!
Badge:
Medium Race
Lake Ladoga Medium Race
Sunday 29th October - 13:00 BST / 14:00 CEST / 05:00 PDT / 21:00 JST PDT / 13:00 JST
Est. Race Distance: 15NM
Est. Race Time: 3 Hours
Weather report: 🌧️ More detail With strong winds all over Europe we're taking shelter in this Russian Lake near St Petersburg.
Short Race
Wŏnsan Short Race
Sunday 29th October - 04:00 UTC / 06:00 CEST / (Saturday 28th) 21:00 PDT / 13:00 JST
Est. Race Distance: 5NM
Est. Race Time: 50mins
Weather report: ⛅ More detail This week we're off the North Korean coast, near the port of Wŏnsan.
Share your progress on Twitter and Facebook, and the Steam Forums - we'd love to hear what you think and how you got on!
Enjoy and safe sailing!
The Sailaway Team
Weekend Races - A race down the Estonian coast
Hello Pioneers,
Here are this weekend's races! Don't forget that in addition to these three races, the first leg of the Volvo Ocean Race is taking place in Sailaway too!
Remember, if you don’t want to sail alone, you can look for a crew in the Multiplayer Forum, there are plenty of sailors looking for crew members or boats to sail with. Perhaps some of you more experienced sailors could team up with newbies and spread the knowledge?
Long Race
Kihnu Island Long Race
Saturday 21st October - 10:00 UTC / 12:00 CEST / 03:00 PDT / 19:00 JST
Est. Race Distance: 120NM
Est. Race Time: 24 hours
Weather report:⛅ More detail A race down the Estonian coast from the island of Osmussaar to the island of Kihnu. The weather is very changeable so keep an eye on the forecast!
Badge:
Medium Race
Karaginsky Island Medium Race
Sunday 22nd October - 04:00 UTC / 06:00 CEST / (Saturday 21st) 21:00 PDT / 13:00 JST
Est. Race Distance: 15NM
Est. Race Time: 3 Hours
Weather report: ☁️ More detail A loop around the northern peninsula of this protected island.
Short Race
Toronto Short Race
Sunday 22nd October - 20:00 UTC / 22:00 CEST / 13:00 PDT / (Monday 23rd) 05:00 JST
Est. Race Distance: 5NM
Est. Race Time: 50mins
Weather report: ☀️ More detail This week we're near the Canadian city of Toronto.
Share your progress on Twitter and Facebook, and the Steam Forums - we'd love to hear what you think and how you got on!
Enjoy and safe sailing!
The Sailaway Team
Sailaway Update V.0.9.021 - A quick hot fix
Hello Pioneers,
Following yesterday’s update, here’s a small update to fix some issues not quite fixed in the last update plus some new additions.
Fixes & Improvements
Switching boats should no longer cancel a challenge or race.
Fixed an issue where improved challenge times were not posted to the leaderboards.
Foresail Sheet and Halyard hotkeys should be fixed (again).
Adjusted foam colour so that it no longer looks blue on browner waters.
Mercury, Venus, Mars, Jupiter, Saturn, Uranus and Neptune have been added to the night sky.
Fixed an issue where the moon would sometimes not appear.
Added a further 55,000 lights, buoys, wrecks and other obstacles along the US coast.
Fixed an issue where lights were not mounted correctly on buoys and did not flash.
Fixed an issue where line graphics would stop updating.
Community Races - The Volvo Ocean Race 2017-18 Leg 1 by elpatron
Hello Pioneers!
We’re excited to share what is becoming Sailaway’s biggest race to date!
Elpatron, one of our most dedicated race creators, has set up a huge race to mirror this year’s Volvo Ocean Race. Since 1973, the Volvo Ocean Race has provided the ultimate challenge to the world’s greatest sailors and is often described as the longest and toughest professional sporting event in the world.
The key details of the race are below but to join the discussion and for information and registration please visit the dedicated race thread on the community races forum. Currently there are over 40 captains registered to take part but you can also drop in on the day!
The Volvo Ocean Race 2017-18 Leg 1
Sunday 22nd October – 14:00 BST / 13:00 CEST / 06:00 PDT / 22:00 JST
Est. Race Distance: 1380NM
Est. Race Time: 9 Days The first leg of the Volvo Ocean Race takes place from Alicante to Lisbon and the Sailaway race will run parallel to the real race!
You can find the race under Activities/Races/Player Created Races in the in-game menu.
You can also join the race's Discord channel here: https://discordapp.com/invite/QmVDveT
Screenshot credit goes to FrankBlack78
As you may have seen in yesterday’s update, we specifically addressed online stability with large numbers of boat so that this race will be the best it can be! See the full details of yesterday's update here.
Make sure to share your progress on Twitter and Facebook, and the Steam Forums - we'd love to hear what you think and how you got on!
Happy sailing!
The Sailaway Team
Sailaway Update V.0.9.020 - Performance updates & a new horizon!
Hello Pioneers,
This update focuses on an issue we believe is the main cause of players crashing when a large number of boats are on screen at once, with a particularly large community race to celebrate the Volvo Ocean Race on the horizon we want to get these performance improvements in place to make this as smooth as possible.
Performance Improvements
Multiple improvements have been made to how Sailaway handles textures and memory allocation, this should stop the root cause of the crashes people are experiencing.
Alongside this improvement there has also been work done to speed up the various background tasks Sailaway computes while you sail, these refinements should lead to better performance on certain hardware.
Horizontal Horizon
We know this is something that has been bothering many of you so we have adjusted the horizon so that it straightens up after a short time, this should hopefully make playing a bit more comfortable.
Weekend Races - Shallow waters in the East Siberian Sea
Hello Pioneers,
Here are this weekend's races!
Remember, if you don’t want to sail alone, you can look for a crew in the Multiplayer Forum, there are plenty of sailors looking for crew members or boats to sail with. Perhaps some of you more experienced sailors could team up with newbies and spread the knowledge?
Long Race
Anzhu Islands Long Race
Saturday 14th October – 03:00 BST / 04:00 CEST / (Friday 13th) 19:00 PDT / 11:00 JST
Est. Race Distance: 210NM
Est. Race Time: 32 hours
Weather report:☁️ More detail A race around the freezing Siberian island of Anzhu, watch out for running aground as some of the waters are very shallow.
Badge:
Medium Race
Santos Medium Race
Sunday 15th October – 21:00 BST / 22:00 CEST / 13:00 PDT / (Monday 16th) 05:00 JST
Est. Race Distance: 15NM
Est. Race Time: 3 Hours
Weather report: 🌧️ More detail A nice course that ends in the Brazilian port of Santos.
Short Race
Perpignan Short Race
Sunday 15th October 13:00 BST / 14:00 CEST / 05:00 PDT / 21:00 JST
Est. Race Distance: 5NM
Est. Race Time: 50mins
Weather report: ☀️ More detail This week we're near the French city of Perpignan.
Share your progress on Twitter and Facebook, and the Steam Forums - we'd love to hear what you think and how you got on!
Enjoy and safe sailing!
The Sailaway Team
Sailaway Update V.0.9.019 - An increase in weather accuracy and more fixes!
Hello Pioneers,
This update brings a host of fixes alongside an increase to the resolution of the weather data currently being used in the game.
Weather Update
We have increased the resolution for the weather data in Sailaway from 1 degree to 0.25 degree which should make for more accurate locational weather. In addition, the white wind lines on the map should now better reflect the true wind direction.
Additional Buoys & Lights
As part of the process of increasing height map data for areas of the world based on extra real-world chart data and extra 3,900 buoys and lights have been added to US waters. This will increase further as more data is processed, the actual height map data will change in a future update.
Boats
Bow wave of the 50' performance cruiser has been reduced
Bow wave angle was has been corrected
Bow spray particles have been reduced as the hull raises from the water. This was especially an issue on the windward hull of the catamaran
Races and Multiplayer
The race panel should now appear for all crew members
When you join a race while there are crewmembers on board, the crew will receive the update and see the race panel + buoys
When you change waypoints while there are crewmembers on board, the crew will receive the voyage updates
When auto navigation is set to maintain angle to the wind and you join another crew, the boat will no longer try to keep steering to the old angle
Auto navigation options are now disabled when the crew member does not have control over the boat
It is now possible to change the auto navigation options if you don’t own the boat
Further improved multi-crew synchronisation
Other changes and fixes
Pressing the Space/Enter key will proceed on any OK or Next prompt that is present
Dolphins were appearing a bit too often, they spawn rate has been reduced a bit.
Fixed another instance of the map not loading correctly
Fixed odd Gennaker rope issue on the Performance Cruiser
Fixed hotkeys not being assigned correctly on the Performance Cruiser
Fixed issue where the furthest right option on a carousel menu could not be selected
Fixed the white texture that appeared on your boat if you had no badges to selected to display.
Here are this weekend’s races – keep your suggestions rolling and let us know where you’d like to race.
Long Race
Virgin Islands Long Race Saturday 7th October – 18:00 UTC / 20:00 CEST / 11:00 PDT / (Sunday 8th) 03:00 JST
Est Distance: 165 NM
Est Time: 33hours
Weather report
Race from the US Virgin Islands, around St Martin and on to Nevis in the Caribbean.
Medium Race
Hell-Ville Medium Race Sunday 8th October – 12:00 UTC / 14:00 CEST / 05:00 PDT / 21:00 JST
Est Distance: 15 NM
Est Time: 3 hours
Weather report
A race around the Madagascan island of Nosy Ambariovato, which way around will you go?
Short Race
Darwin Short Race Sunday 8th October – 04:00 UTC / 06:00 CEST / (Saturday 7th) 21:00 PDT / 13:00 JST
Est Distance: 5 NM
Est Time: 50 mins
Weather report
This week we’re off the coast of Darwin in Northern Australia.
Share your progress on Twitter and Facebook, and the Steam Forums - we'd love to hear what you think and how you got on!
