Documentation

How atp2osm works, and how to contribute.

What is a “brand”?

The notion of a brand comes straight from All The Places (ATP). ATP is a collaborative database built around national and international chains: retail chains, restaurant networks, banks, pharmacy chains… Every chain is identified there by a unique Wikidata code, for instance Q37158 for Zara. Independent shops (the bakery around the corner, a local restaurant…) are not part of it.

A few key terms

spider: A program dedicated to a single chain, which automatically collects its data from its official website (opening hours, phone, address…).
scraping: The technique of automatically extracting information from web pages. It is what every ATP spider does to gather the data of a chain.

atp2osm follows that structure and integrates the data chain by chain, following the ATP spiders closely. This guarantees that all the data of one chain comes from a single, homogeneous source, which is what makes it possible to judge the quality of a spider by reviewing a random sample of 3 POIs per integrated batch, and to extend that confidence to the whole dataset.

A chain is missing? Its ATP spider may not exist yet.

Report it in the missing brands Write a spider on GitHub

Summary

What it does

Add the missing opening_hours, phone, website and email tags
Match nodes, ways and relations
Group the integrations by subdivision
Keep a full trace in the history

What it does not do

Create new OpenStreetMap nodes
Change or delete existing tags
Change the geometry
Integrate without a human review

How it works

atp2osm enriches OpenStreetMap from the All The Places (ATP) data. ATP automatically collects the information of points of interest (opening hours, phone, website, email) straight from the official websites of the chains.

The tool compares that data against the existing OpenStreetMap nodes, finds the missing tags, and lets you review then integrate them into OpenStreetMap through your own account. No node is ever created or deleted: only missing tags are added.

The process in detail

Collecting the ATP data

Every week, the latest ATP data is downloaded from alltheplaces.xyz. It holds the POIs with the attributes extracted from the official websites of the chains.
Refreshing the OpenStreetMap data

In parallel, the OpenStreetMap data is imported into the local database through osm2pgsql.
Spatial and attribute matching

Every ATP POI is matched against the OpenStreetMap nodes within 500 m, checking that at least one attribute is shared among brand:wikidata, brand, name, phone, website, email. Ambiguous matches (several OpenStreetMap nodes for one ATP POI) are dropped.
Computing the missing tags

For every match kept, the tool computes which tags exist in ATP but not in OpenStreetMap. Only the missing ones are offered: an existing tag is never changed.
Review and integration

You check the changes offered for a brand, then you trigger the automatic integration of the whole dataset. The changes are sent to the OpenStreetMap API through your account, grouped by subdivision.

How to contribute

1

Sign in with your OpenStreetMap account

Use the sign-in button at the bottom of the menu. Your integrations are recorded under your name in the OpenStreetMap changeset history.
2

Pick a brand

The Brands to integrate page lists every chain for which enrichments are available, with the number of nodes concerned.
3

Check the matches

For every node, check that the ATP ↔ OpenStreetMap match is right: location, name of the place, tags offered. When in doubt, open the ATP source or the OpenStreetMap object before accepting.
4

Integrate

If the data looks right, start the integration. The changesets are created automatically on OpenStreetMap, one per subdivision. The operation cannot be undone: make sure you have checked the data.

Reviewing a brand

At least 3 items per batch are reviewed by hand, drawn at random from the places in that batch.

The review goes batch by batch: a brand is cut into batches of whole subdivisions, and before integrating a batch you check a sample drawn at random from it. If that sample is right, the data is considered reliable for every place of the batch.

The sample holds at least one example of every tag the batch changes: if the batch adds, say, phone, website, opening_hours and email, the sample will hold a POI carrying each of them. Every tag is therefore seen at least once before the integration, and the sample is filled at random up to 3 items when it holds fewer. Depending on the tags involved, you may well have 4 POIs to check, or more.

Before accepting, make sure the items you checked are representative: try several places in different areas, look at the edge cases (unusual opening hours, badly formatted phone numbers, redirected URLs…). If the slightest doubt remains about the quality of the spider, report an error rather than accept.

Integration statuses

Every integration recorded in the history is given a status reflecting its outcome.

Success

Integration succeeded

Every change has been sent to OpenStreetMap and is now publicly visible.

Partial

Integration partly succeeded

Some subdivisions could not be integrated because of an OpenStreetMap API error. The changes that were sent are visible on OpenStreetMap all the same. The data left out stays available for a later integration.

Cancelled

Integration cancelled, data reported

The integration was stopped after a contributor found errors in the ATP data. Nothing was sent to OpenStreetMap.

Error

Technical error

An error unrelated to what you did stopped the integration. Nothing was sent to OpenStreetMap. You can start the integration again from the page of the brand concerned after 4 weeks. That delay leaves time to fix the errors the datasets may hold.

Reading the spider figures

The spiders page counts what each All The Places spider brings, step by step.

Scraped here	The POIs the spider produced inside the country and its territories.
Matched	Those with an OpenStreetMap object within 500 m sharing a brand, a name, an email, a phone or a website. The rest is either missing from OpenStreetMap, or present without any attribute to recognise it by.
To integrate	The matches with something left to do: a tag to add (wave 1) or to replace (wave 2), minus the subdivisions under cooldown. A POI on both waves counts twice, as two integrations. The brands page shows one wave at a time and only the brands carrying a Wikidata code, so its totals are lower.
Integrated	The POIs actually written to OpenStreetMap through the brands the spider produces; a brand two spiders share counts for both.

Licences

OpenStreetMap

The OpenStreetMap data is published under the ODbL 1.0 (Open Database License). Every contribution made through this tool falls under the OpenStreetMap Contributor Terms.

All The Places

The ATP data is published under CC0 1.0 (public domain). It can be used freely, without restriction.

atp2osm

The source code of this tool is published under the GPL-3.0 licence. Contributions are welcome on GitHub.

Exporting the data

The data produced by atp2osm can be exported freely, as CSV or JSON, without signing in. It is only the trace of the integrations and the list of the missing brands: the source data stays with the projects that produce it (All The Places and OpenStreetMap).

GET/api/export/history.csv: One row per integration: brand, contributor, date, status, number of objects and of tags.
GET/api/export/subdivisions.csv: The detail per changeset: administrative subdivision, number of objects, OpenStreetMap changeset identifier and status.
GET/api/export/todo.csv: The brands reported as missing from ATP. The ones ATP now knows are left out, unless ?show_in_atp=1 is passed.
GET/api/stats.json: The figures of the statistics page: totals, series per period, top tags, brands and contributors, spider reliability. Takes the same filters as the page.

Replace .csv with .json for the same content as JSON. The history and todo exports take the same filters as their pages (q, status, user, from, to); the “Export” button above each table carries the active filters over anyway.

Resources

All The Places — the source of the POI data
OpenStreetMap wiki page — documentation of the automated integrations
Source code — contributions welcome