The next stage in my Google Maps experiments has just gone live. We're now plotting all Ecademy subscribers on a Google Map using my automated bunching code. I think we're the first social or business network to do something like this.

If anyone's worried about data security, we're only showing members who allow their street address and postcode/zipcode to be shown on their profile.

The Geocoding was done from an open, non-commercial database of the first part of people's UK postcodes or US Zipcodes. They can then fine tune this.

The infowindow popups contain a link to their Ecademy Profile, Company web site and their main IM system, such as Skype. The Skype links are active and show their current status if they've enabled "Show my status on the web". I'm also using the latest Google Maps V2.x to show tooltips when you hover over an point.

Here's a fun punctuation variation in a post from Uncle Dave about the Da Vinci code.

Yeah it was worth seeing, if only to understand how dead, popular culture is.

Yeah it was worth seeing, if only to understand how dead popular, culture is.

Anyway. Just before that are two great posts about RSS and the dangers of centralising a decentralised technology. And it does amaze me when I see Typepad, Wordpress and Blogger users using Feedburner. So it's equally puzzling to see yet another feed processing system.

I'm attempting to Skypecast the ecademy event tonight. Details here.

https://skypecasts.skype.com/skypecasts/skypecast/detailed.html?id_talk=5230 [from: JB Ecademy]

ProgrammableWeb.com » Blog Archive » Invasion of the Robots Contest

Skype should do this.

More work on automatic bunching.

I've now sussed working across the international dateline. The problem was that the dataport coordinates were ending up with longitude either >180 or less than -180. To get the wrapping right, I needed to either subtract or add 180 respectively.

Official Google Base Blog: Create ads for your itemsIf you're familiar with Google AdWords, you'll be happy to know that we're now enabling you to create AdWords ads for individually posted items in Google Base, right from the Google Base edit item page.

Very interesting. I've thought for a while that eBay should get into this business. Create Ads for a fee from eBay listings and then place them context sensitively on blogs. Of course, Google is better placed to do this. The next question is whether there's an API so you can do this automatically when you create the Base entries. Anything to improve the quality of Google Ads is welcome. As is anything that increases the inventory of much more specific Ads. I'm sick to death of "Node. Get best prices on Node at eBay" just because my main page is called node.php

Let's think about this a bit further. eBay could really benefit and pass that benefit on to their posters by teaming up with Google to automatically create Adwords Ads from eBay listings. And they should be able to get their posters to pay for it as well rather than spending huge amounts on generic Ads.

We'll even geo-target your ads to the most appropriate location. This is also significant and begs the question of just how good Google is at getting the Geo coordinates of the viewing browser. I'd really like to see this worked in with a service like Plazes that allowed the users to specify exactly where they currently are as well as just working off the IP address. If that's what they're doing.

More work on bunching and display in Google Maps.

I've currently got both a "Viewport" in Google Maps and a "Dataport". The Viewport is what you can see, the dataport is the area for which I get points to plot when I do a refresh. The dataport is twice the size (4 times the area) so that if you scroll the map a short way I don't need to do a round trip to get new data.

What I'm now doing with bunching is this. I've divided the dataport into an 8*8 grid, which implies a 4*4 grid for the viewport. I get the data for the whole dataport. If there are more than 64 points in total I iterate through the cells of the 8*8 grid. If a cell has more than 2 points, I mark the first one as a bunch. I take the average of the others and compute a centre point for the group and apply it to the first point. I then mark the others as "deleted". When I've gone through all 64 cells, each cell has at most 2 points so in the worst case there are 128 points to be mapped. I then go through the remaining points and if there are 2 with identical locations I spread them slightly by adding a small random offset.

The next problem is dealing with the international dateline where the viewport and dataport straddle the transition from longitude 180 to -180. The best solution I've found so far is to work out which side has the most visible and to ignore the other side. So the dataport edge is fixed at 180 or -180. This seems to work well except at one zoom level and map centre. I think this where the dataport actually straddles both 180 and zero. Still working on this.

Apart from the dateline problem there's one last glitch. This happens where there are more than 64 points at a single location. In my dataset this almost happens for "London, UK". The problem is that no matter how much you zoom in, there will always be more than 64 points in total and they are all in one cell so will be bunched. The other limitation is going to be dealing with very large numbers of points, say bigger than 10,000. At some zoom and centre values, bringing all the data into an in memory array and iterating though it repeatedly is going to slow down too much. At that point it would probably be more efficient to do 64 SQL queries rather than one large one and then 64 runs through the data.

So I'm closing in on a set of algorithms to automatically bunch points and to keep speed up by limiting the total number of points that are actually mapped. It works really well!

There's a new beta of Live Messenger 8.0.


Why have all the major IM suppliers forgotten how to ship early, ship often? Skype is shipping a major release every qtr. The rest of them have a release schedule measured in years. The most annoying is Googletalk. It's now 8 months since the LibJingle announcement. Where's the code that supports it? Where's the next big release, because it still looks like Alpha code.

None of them (with the possible exception of AOL) seem to understand that voice, video and conferencing (both chat and voice) are important. Skype is stealing a major march on them here. And what about the 3rd parties. Where's Trillian and Gaim's voice and video?

And none of them really understand the importance of an API and things like showing status on the web with a simple URL scheme to launch chat/voice.

So this beta was good because it finally lets you send a message to somebody who's offline, delivered when they next appear. And it was good because it prompted an upgrade automatically. But it was bad because the install required a reboot of XP. *WHY?* And the shared folders is clever. but why is there still all the adverts and that stupid "Billing Information" menu choice. And did they finally lift the 150 contacts limit on the directory size? And does voice and video actually work? I use Skype Voice every day. But I almost never use MSN voice and video because it hardly ever seems to work.

I've spent some more time on my Skype to Google Maps prototype. The main challenge has been finding ways to make it more responsive. And the big problem is that Google Maps slows up dramatically when plotting more than about 50 points. So here's the basic setup

1. Use the Skype API via Delphi to extract all the contacts and their profile information.
2. Upload this to a PHP page whihc stores it in a mysql database.
3. Produce a php page that produces an XML file suitable for the Google API AJAX input based on the SW and NE corners of a square. It takes swlat, swlong, nelat and nelong as parameters. Does a Mysql search for entries with lat/long within that square and returns the results.
4. A Google Maps page that does the basics and then gets its data from the php search based on the current map viewport.

There's about 500 points in the database. This is preparation for when I've got 10,000 points. Even with 500 I can't just display them all at once because the initial paint is way too slow. So I need to refresh the points when the map changes.

So the first question is when to refresh the data. If you refresh on every move using the moveend event, you get a wait for new data even if you only move a few pixels. Plus, popping up an info window will auto-move the map if it's close to the edge. So what I do now is plot for a space that is 4 times as big (a border of half a map width in each direction. NSEW). If at the end of the moveend, the viewport is still within this space don't bother to refresh. I've then got a toggle using zoomend to force a refresh when the user changes the zoom level.

Now at lower zooms you can still get more than 50 points needing plotting. I need a way of bunching the points together when they're close. I did a search for people doing this and found this page. Now he suggests doing the work in the database to bunch points and then reading it out depending on the zoom level. This probably produces the neatest results but it's a lot of extra work. I wanted an algorithm that would bunch automatically.

So in the backend XML page, I do the basic query and get the results into an array. I then iterate through the array in a two layer loop looking for points that are within 1/10th of the map width and 1/10th of the map height. When I find one, I discard the point I'm exploring and alter the type of the matching point to "bunch". I then compute the point half way between them and change the matching point's lat/long to this mid point location. This effectively divides the map into 100 squares and then aggregates points so that there's no more than one in any square. If there are two or more, they're replaced with one "bunch" point. Now we have a maximum of 100 points returned. Finally I read through the array and generate the XML whihc looks like this.

<marker lat="51.2091125" lng="-0.78744615625" type="bunch">
There's a bunch of 7 near here.
Click to zoom in and see them.
</marker>

<marker lat="51.1289" lng="-0.015761" type="point">
Some HTML for the infowindow
</marker>

The first is a bunch point, the second is a real point. The contents of <marker></marker> is the html to be displayed in the infowindow.

So now we've got the map displayed with it's points. And it only refreshes the points when you move half a map away or zoom. And we've limited the number of points plotted by bunching them.

I use a standard icon for the main points and a custom icon for "Bunch" points created by recoloring one of Googles push pins.

The last stage is to handle events for the UI. For plain points, I just use Marker.click to popup the infowindow using the html I passed over in the XML. I've also added a Map.click which checks if a marker has been clicked if it hasn't, then the user clicked in space between points and I close any open infowindow. For bunch points, I've also added a marker.mouseover and marker.mouseout which popup the infowindow explaining what a bunch point is. Finally, there's a bunch marker.click which zooms in, and then centres the map on the bunch point. This last caused me some problems. zoom also does a move and then the centring does a move. I had to be quite careful about sequence and added an extra flag so that the old points do actually get cleared before the new points get drawn.

The next problem is when 2 or more points have exactly the same location. Perhaps because the geocoder was limited or because people live in the same town and that's all the address I've got. What I did was to add some small randomness (1/100th of map width) to spread the points out a bit.

So there we have it. At the cost of some php processing in memory I can now handle large numbers of points with a fairly simple and self explanatory bunching algorithm. It seems to work well at small to medium zoom levels but still needs some fine tuning for country level maps.

Now here's a complete aside. The UK postcode data is proprietary, commercial and expensive. There are free databases of the first part of the code to lat/long but not to any deeper level. So how about we build a database of the full postcode to lat/long. We then get people to enter their first part, draw a Google map based on that and then get them to locate their actual location on the map. We can then read out the lat/long and store it in the database. Over time we'd build a free and open dataset of full postcode to lat/long. We then offer a REST API to geocode the full postcode. Anyone up for this? email me at julian_bond at voidstar.com The usual problem applies. How do we give enough immediate value back to get people to actually do this and enter the data? And how do we clean the data and make sure it's reasonably accurate.

And finally, the Skype Map mashup is an example of the limited security model around this data. Someone authorises me to see their profile. I can then get their address, geocode it and plot it and then present it to the world with no further authorisation from owner. Now I don't see how Skype can prevent this as long as they allow me access to the data.

Left this on Doc's weblog.

The key phrase is "a government-managed regulatory habitat".

Maybe what is needed is more legislation, not less. And not controls over what the Telcos and cable companies do with net neutrality but controls to force them to open up their monopoly to competing 3rd parties. eg

- Force them to sell wholesale bandwidth to 3rd party ISPs - Force them to sell space in their switching centres to 3rd party ISPs to unbundle the local loop.

When direct competitors are selling net neutral broadband, how will the Telcos be able to offer hobbled broadband?

The problem here is a common one to all utilities that have a monopoly hold over a single connection on the last mile. Phones, electricity, gas, water, sewage, cable TV, etc. The government has to control this for the common good. But the government can do this by granting a monopoly with controls or by creating an artificial market with competition. If they do the second, then they have to fine tune it to reflect changes in technology and possibilities. If they don't, it just falls back into the first approach. So we have the absurd situation of AT&T being broken up into the Baby Bells only for them to then merge back into a new "AT&T".

Before calling for Net Neutrality legislation take a long hard look at the UK experience and what the UK has done with BT. It's not perfect, but it almost works. BT had a similar government managed monopoly to the US Telcos. They've been forced to sell wholesale broadband and LLU. Now we have a genuinely competitive market that is forcing up broadband functionality and forcing down broadband cost.

And finally, the Cable companies have had it easy for too long. It's now time to treat them the same as Telcos. They got lots of tax credits for laying cable. It's now time to treat that cable as an unfortunate monopoly over a last mile and force them to open it up to 3rd parties.

He replied with some agreement and an argument that basically said "It's different in America." At whihc point too much coffee led me to this.

> Also impossible. The telcos, like the copyright giants, have > successfully positioned their holdings as "simple property". They > built the infrastructure. It's their property. Why shoud government > come in and tell them how they can and cannot make money with that > property? That's anti-business, anti-market, etc. Won't fly.

I know you know this. And I know it is different in the USA and maybe impossible.

But "anti-business, anti-market" is doublespeak. There is no market when the last mile is controlled in a government-managed regulatory habitat unless the government also creates an artificial market. The government has created a regulated monopoly. Now it needs to create a regulated market.

Looking from the outside at the USA it feels like we see this constantly. Endless bleating about pro-business, pro-market, pro-capitalist, anti-socialist, freedom of this that and the other, when reality is a very highly regulated, very highly government controlled environment where the government is for sale to the highest bidder (sorry, lobbier). If it wasn't for the last bit about everybody working for their shareholders (government included) it would be socialism. It's not capitalism. There isn't really a word for it. Unless it's Pigopolism. ;-)

Completely free. Just as long as you stay between the white lines on the freeway. And if you have enough money you can tell the government where to put the white lines.

Sorry. I seem to have gone off on one there...

Seems my Anti-US sentiments are getting the better of me again :) It also seems that I keep on having these conversations where I point out an alternative and the other side is unwilling to even consider it because "That's just the way it is". Now I'm not specifically accusing Doc here of closed minded thinking but something much more general. I do think a lot of people and in particular USians on the web seem to be unable to see past the status quo and to think that there might be an alternative. It doesn't have to be this way. It really doesn't. The existing way of things may have huge momentum but it can and does change.

Whatever. As a Brit I really should stop bothering with US Net Neutrality issues. If the typical US broadband customer gets screwed by their monopoly Telco ISP will it actually affect me at all, at all? Won't it just mean that the centre of innovation will move a bit more out of the USA?

Spring and early summer are finally upon us because the swifts have arrived. yay!

The engineering work yesterday took longer than expected due to a collection of small unforeseen problems. However we are up and running in our new rack.

We still have some work to do today (Sunday April 30) which may mean some short outages. Please bear with us, if Ecademy is out of action today. [from: JB Ecademy]

popurls.com | popular urls to the latest web buzz
Where is the internet's attention focused today? [from: del.icio.us]

Dave W had an entry yesterday about BT and needs.

We need some BT help on the publishing end. Blog software has got quite good at one click publishing of rich media files and then automatically adding them to the RSS/Atom feeds. We need to make BitTorrent publishing as easy and simple. There's work to be done here to integrate the sort of approach at http://www.blogtorrent.com/ into mainstream blog software. Critical, I think, is to:-

- Hide the complexity of creating the torrent
- Having the web end be a BT client and automatically seed so there's always a full seed available
- Having the client end stay online and guarantee 100% sharing from everyone who downloads.

Wordpress developers! Are you listening? How about Drupal?

BTW. Did we solve the problem of constructing a podcast feed that contains both plain and BT enclosures of the same file?

Skype Journal: Two phreaks experiment with Skype contact integration :

Skype is relatively immune from SPIM because a lot of Skype users have their privacy settings set to only receive chat and voice from people in their contact list.

I was asked to do a Skype API program to blast the same message to all of a person's contacts. It's quite easy to do and I did a proof of concept that takes a message from a form sends it and then closes each message window that pops up. you can then deal with the multiple popping windows as people reply at your leisure. But I didn't give it to the person who asked because I didn't want to be on the receiving end.

Apart from scouring the web for addresses, the other approach is to build an API prog that people want to use (like say put winamp listening to into your mood). And then report their contact's profiles up to a database. This is getting close to a viral technique for gathering profile data. It's not necessarily evil but it's pushing the boundaries.

Home Taping is Killing Music - Wikipedia, the free encyclopedia

Yay! My image made it into Wikipedia as an example of a parodied version.

Enjoy. The next stage in the proof of concept is now done.

1. Extract Profile information using Skype API, Delphi and the Skype4Com Activex


2. Upload the data to a Mysql dbms by calling a php page



3. Convert "City, Province, Country" to Lat/Long using The Worldkit Geocoder and some (ahem!) by hand with Google Maps.


4. Display the Map and fill in the points with the Google Maps API. This calls a PHP script to get entries from the DBMS based on the lat.long rectangle of the current map.


5. Any Questions? julian_bond at voidstar.com or skype:julian.bond?chat

Notice the small ads top right. They pay good money. Still chicken feed but better than AdSense.

But what have I done?

It looks to me like the benefit to the advertiser is taking advantage of my pagerank. Which means I'm distorting Google.

Have I become Evil?

I've currently got 534 contacts. Of these 257 have got good enough address detail to be able to get a single location in Google Maps. Skype is reporting that 233 of those contacts were last online this year. The intersection of the two is 67 people online this year and with address detail.

Now I've been collecting Skype contacts since the very early days so I may have a disproportionate number of old and dead entries. But a rule of thumb suggests that maybe 25% of someone's contact list can successfully be mapped.

Which then raises the question. Why don't people fill in their profiles on Skype? And when they do, why don't they fill them in accurately? There's 117 entries online this year but with blank country.