I've spent some more time on my Skype to Google Maps prototype. The main challenge has been finding ways to make it more responsive. And the big problem is that Google Maps slows up dramatically when plotting more than about 50 points. So here's the basic setup

1. Use the Skype API via Delphi to extract all the contacts and their profile information.
2. Upload this to a PHP page whihc stores it in a mysql database.
3. Produce a php page that produces an XML file suitable for the Google API AJAX input based on the SW and NE corners of a square. It takes swlat, swlong, nelat and nelong as parameters. Does a Mysql search for entries with lat/long within that square and returns the results.
4. A Google Maps page that does the basics and then gets its data from the php search based on the current map viewport.

There's about 500 points in the database. This is preparation for when I've got 10,000 points. Even with 500 I can't just display them all at once because the initial paint is way too slow. So I need to refresh the points when the map changes.

So the first question is when to refresh the data. If you refresh on every move using the moveend event, you get a wait for new data even if you only move a few pixels. Plus, popping up an info window will auto-move the map if it's close to the edge. So what I do now is plot for a space that is 4 times as big (a border of half a map width in each direction. NSEW). If at the end of the moveend, the viewport is still within this space don't bother to refresh. I've then got a toggle using zoomend to force a refresh when the user changes the zoom level.

Now at lower zooms you can still get more than 50 points needing plotting. I need a way of bunching the points together when they're close. I did a search for people doing this and found this page. Now he suggests doing the work in the database to bunch points and then reading it out depending on the zoom level. This probably produces the neatest results but it's a lot of extra work. I wanted an algorithm that would bunch automatically.

So in the backend XML page, I do the basic query and get the results into an array. I then iterate through the array in a two layer loop looking for points that are within 1/10th of the map width and 1/10th of the map height. When I find one, I discard the point I'm exploring and alter the type of the matching point to "bunch". I then compute the point half way between them and change the matching point's lat/long to this mid point location. This effectively divides the map into 100 squares and then aggregates points so that there's no more than one in any square. If there are two or more, they're replaced with one "bunch" point. Now we have a maximum of 100 points returned. Finally I read through the array and generate the XML whihc looks like this.

<marker lat="51.2091125" lng="-0.78744615625" type="bunch">
There's a bunch of 7 near here.
Click to zoom in and see them.

<marker lat="51.1289" lng="-0.015761" type="point">
Some HTML for the infowindow

The first is a bunch point, the second is a real point. The contents of <marker></marker> is the html to be displayed in the infowindow.

So now we've got the map displayed with it's points. And it only refreshes the points when you move half a map away or zoom. And we've limited the number of points plotted by bunching them.

I use a standard icon for the main points and a custom icon for "Bunch" points created by recoloring one of Googles push pins.

The last stage is to handle events for the UI. For plain points, I just use Marker.click to popup the infowindow using the html I passed over in the XML. I've also added a Map.click which checks if a marker has been clicked if it hasn't, then the user clicked in space between points and I close any open infowindow. For bunch points, I've also added a marker.mouseover and marker.mouseout which popup the infowindow explaining what a bunch point is. Finally, there's a bunch marker.click which zooms in, and then centres the map on the bunch point. This last caused me some problems. zoom also does a move and then the centring does a move. I had to be quite careful about sequence and added an extra flag so that the old points do actually get cleared before the new points get drawn.

The next problem is when 2 or more points have exactly the same location. Perhaps because the geocoder was limited or because people live in the same town and that's all the address I've got. What I did was to add some small randomness (1/100th of map width) to spread the points out a bit.

So there we have it. At the cost of some php processing in memory I can now handle large numbers of points with a fairly simple and self explanatory bunching algorithm. It seems to work well at small to medium zoom levels but still needs some fine tuning for country level maps.

Now here's a complete aside. The UK postcode data is proprietary, commercial and expensive. There are free databases of the first part of the code to lat/long but not to any deeper level. So how about we build a database of the full postcode to lat/long. We then get people to enter their first part, draw a Google map based on that and then get them to locate their actual location on the map. We can then read out the lat/long and store it in the database. Over time we'd build a free and open dataset of full postcode to lat/long. We then offer a REST API to geocode the full postcode. Anyone up for this? email me at julian_bond at voidstar.com The usual problem applies. How do we give enough immediate value back to get people to actually do this and enter the data? And how do we clean the data and make sure it's reasonably accurate.

And finally, the Skype Map mashup is an example of the limited security model around this data. Someone authorises me to see their profile. I can then get their address, geocode it and plot it and then present it to the world with no further authorisation from owner. Now I don't see how Skype can prevent this as long as they allow me access to the data.

[ << A rant on Net neutrality ] [ MSN Messenger Beta >> ]
[ 07-May-06 5:47pm ] [ , ]