Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains all the map data and you can request ranges from the single file to get the required vector data you need for a given area/zoom.

So you can setup something like the following:

    S3 (host the myfile.pmtiles) <- Lambda (takes the x/y/z from the path and requests the correct range) <- Cloudfront cache tile response 
Then you can setup tiles.mydomain.com (or use the cloudfront domain directly) and then use Leaflet or similar on the frontend to fetch/render the tiles. For Leaflet you use the protomaps plugin/lib and give it a url like "https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt" where "20230408" maps to "20230408.pmtiles" in your S3 bucket. Now I can drop new pmtiles files into that bucket and update my clients to use the new source. And since the tiles are in vector format you can theme them however you want in the client which is neat. Lastly you don't have to use the 100+GB whole-earth tileset. You can use a tool [1] (provided by the same guy) to download a dataset for just a given geographical region.

The .pmtiles file is a little over 100GB but the whole setup took me only an hour or two max to get running and will cost way less than Google Maps to run.

[0] https://protomaps.com/

[1] https://app.protomaps.com/downloads/small_map



their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution

digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer


I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data.

Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map which is actually better.


I just updated https://protomaps.com it to use the more detailed file you found, thanks. So instead of a zoom level of 12 it should work down to 20+ - the actual dataset contains up to zoom 15, which is ~1 billion tiles in a single 110GB file.

It's important to me that I showcase the system working as advertised - you are free to download the 110GB file from that static storage with no restrictions. Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Before, it was on DigitalOcean Spaces, which is low latency but would cost me $1-2 out of my own pocket each time someone clicked the link.

The Lambda/Workers integration as described in the other comment is the preferred solution to this for developers operating this at planet scale. Of course, the static storage solution is perfect for small to medium-sized areas, even on GitHub Pages.


I just wanted to say thank you for everything you've done for Protomaps. It was very easy to get started with and other than a small Vue/ES6-related issue I was able to get it integrated into my codebase very quickly.


> Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees.

Can you expand on this a little? Cloudflare is supposed to be pretty competent as CDN and with caching.


This is Cloudflare's new blob storage product R2. Putting the free Cloudflare CDN and cache infront of R2 doesn't work with HTTP Range headers, likely for product differentiation from their Video CDN. The latency of R2 is also much higher than S3 or GCP, but I know the R2 team is working hard on improving this.


FWIW you can add your own caching logic with Workers, if the CDN for some reason doesn't do Range. The Cache API exposed to Workers is quite useful.


That is exactly how the open source Workers integration works:

https://protomaps.com/docs/cdn/cloudflare

However, it is an optional acceleration layer on top of the PMTiles access pattern; and it doesn't make a compelling demo, it's just a Z/X/Y layer which is how map tiles have worked for the past 20 years.


Hmm, thats cool, but I need to have raster maps pushed to my endpoints. Do you know any easy way to use mvt maps and generate tiles from them on the fly (CGI backend + caching) and then use Leaflet to render those?

I have my own tiny personal project for maps renderer using Leaflet. For now I use 3 sources for tiles (Google, OSM and Esri) but having more sources would be very cool.


I don't personally know how to do that. It appears the PMTiles format does support raster maps but the dataset I use is only vector as far as I know.


So one Lambda execution per tile?


It shouldn’t be required if the range requests are right and you have a compatible renderer in the client.

To expand a bit -- There are adapters for mapbox(gl/libre) and leaflet, and so on that will allow you to use a .pmtiles directly from the client with a range request supporting server, like s3 or nginx.

I've used protomaps basemaps to generate an Ireland and EU basemaps for a project from OSM data. I still haven't quite figured out the tuning for which features should be shown, combined, or excluded yet, and the generated layer names are sort of, but not quite compatible with similar base layers + styles from mapbox (which uses the same base data, but a different, proprietary conversion). This part takes a reasonable skill as a GIS person and some taste in the design of the feature symbology.

The downside of doing this on S3 directly is that you might wind up publishing a link to a 100 gig file that would cost $10 if someone just downloaded it. Lambda invocations seem cheap compared to that risk. OTOH, throwing it on a hetzner box is quite reasonable.


The transformation of OpenStreetMap features into the basemap is open source and available under a permissive license now at http://github.com/protomaps/basemaps

This is the focus of most development for the next few months - like you said, it's also a matter of aesthetics and context, like if the map is underlying another PMTiles data layer it should be lower contrast then if it's overlaid with pin markers. The end goal is to have a flexible basemap solution adaptable to most apps but you are free to roll your own proprietary style on top of the open source tiles if you'd like (there are already companies doing this)


Yes, that's what I was using for my basemaps, it's quite fast for Ireland on a relatively fast machine (minutes), but Europe as a whole was hours. (hetzner quad core 8 thread ~3yr old intel machine with 64G ram)

I've got a couple of goals, or at least evaluations for goals --

1) Have a stripped down basemap like Carto Light, but with different emphasis for the "bicycling friendly" roads, and selectively higher contrast/emphasis for those minor through roads that discourage car traffic but are perfect for a bike. (e.g. Irish boreens, those 1 lane or double track "paved" L roads that are everywhere outside of the major cities.) I find that I simply can't see the minor roads on apple/google maps when I'm out unless I'm zoomed in to the point that I can barely see a network. (fading eyesight and super low contrast).

2) Another lighter base map, but with transit focused features to compliment some of the open data in Ireland around transit -- the routes and timetables and so on. I'm aiming for a site that can be 100% statically hosted but will show the routes in a more friendly manner than Bus Eireann's site.

3) If it goes well, I might be doing this in a more commercial context for some clients who are currently using mapbox for their basemaps but due to political concerns need to insert in different names/boundaries for disputed areas/features. We'd love to be able to self host, but quality is a major concern there.

The bugs that I'm seeing in the 1 and 2 cases are things like handling the look of freeway interchanges/flyovers at close zooms, consistently getting rivers to show at appropriate zooms because they're into two separate feature types depending on the width, with the narrow bits getting dropped. And the general mismatch between the styling I'm used to from the mapbox converted layer names/feature types and what's coming out of pmtiles/basemaps. The feature bits look like tippecanoe coalesce/drop densest features tuning, but I'm also looking at just dropping out entire feature sets to drop the size of the tiles. That should help the coalesce/dropping behavior as well.


Thanks - for the issues related to freeway interchanges and rivers, it would be most helpful to report these on GitHub using screenshots and links to lat/lon positions, or even links to OSM nodes and ways.

OSM is a freeform dataset and not a cartographic product, so most of the basemap work now and in the future is on getting good results of this transformation for 200+ countries. The mismatch vs. existing maps must exist because we need to ensure to downstream commercial users that all end products of the map generation are openly licensed - that's why this is being pursued as an independent, self-funded project with support from GitHub Sponsors: http://github.com/sponsors/protomaps


Ok, I’ll dig a bit more to get some good test cases with the local data that I’m familiar with, and make sure that it’s not something that I’m doing wrong.


Library author here - yes, one execution happens per tile, if you have CloudFront in front those will only happen when the PoP has a miss.

With 512MB RAM these generally complete in under 100ms so the unit costs of Lambda are within a few multiples of Cloudflare Workers (.5USD / million invocations)


Hey, want to say I'm super excited by protomaps. I'm struggling a bit to convert existing OSM pbf files into the pmtiles format. Specifically, I'm trying to convert this guy https://daylightmap.org/

Do you have any tips for doing so without renting memory in the cloud?


The daylight PBFs are difficult to combine without a lot of RAM, there's no way around it right now. Either do that or hold out for potential newer releases of the data delivered as Parquet files.


Like the other comment says I don't think it's quite that bad but also CloudFront (or whatever CDN you use) is going to cache the response. In my case I'm rendering restaurant locations on a map. I only have X number of items that each have a map of the restaurants they are available at so unless people pan/zoom around a lot I only expect a handful of lamba calls and the resulting tile data will be cached in CloudFront for everyone else.




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: