atp2osm logo atp2osm

Documentation

How atp2osm works, and how to contribute.

What is a “brand”?

The notion of a brand comes straight from All The Places (ATP). ATP is a collaborative database built around national and international chains: retail chains, restaurant networks, banks, pharmacy chains… Every chain is identified there by a unique Wikidata code, for instance Q37158 for Zara. Independent shops (the bakery around the corner, a local restaurant…) are not part of it.

A few key terms

spider
A program dedicated to a single chain, which automatically collects its data from its official website (opening hours, phone, address…).
scraping
The technique of automatically extracting information from web pages. It is what every ATP spider does to gather the data of a chain.

atp2osm follows that structure and integrates the data chain by chain, faithfully to the ATP spiders. This guarantees that all the data of one chain comes from a single, homogeneous source, which is what makes it possible to judge the quality of a spider by reviewing a random sample of 3 POIs per integrated batch, and to extend that confidence to the whole dataset.

Summary

What it does

  • Add the missing opening_hours, phone, website and email tags
  • Match nodes, ways and relations
  • Group the integrations by subdivision
  • Keep a full trace in the history

What it does not do

  • Create new OSM nodes
  • Change or delete existing tags
  • Change the geometry
  • Integrate without a human review

How it works

atp2osm enriches OpenStreetMap from the All The Places (ATP) data. ATP automatically collects the information of points of interest (opening hours, phone, website, email) straight from the official websites of the chains.

The tool compares that data against the existing OSM nodes, finds the missing tags, and lets you review then integrate them into OSM through your own account. No node is ever created or deleted: only missing tags are added.

The process in detail

  • Collecting the ATP data

    Every week, the latest ATP data is downloaded from alltheplaces.xyz. It holds the POIs with the attributes extracted from the official websites of the chains.

  • Refreshing the OSM data

    In parallel, the OSM data is imported into the local database through osm2pgsql.

  • Spatial and attribute matching

    Every ATP POI is matched against the OSM nodes within 500 m, checking that at least one attribute is shared among brand:wikidata, brand, name, phone, website, email. Ambiguous matches (several OSM nodes for one ATP POI) are dropped.

  • Computing the missing tags

    For every match kept, the tool computes which tags exist in ATP but not in OSM. Only the missing ones are offered: an existing tag is never changed.

  • Review and integration

    You check the changes offered for a brand, then you trigger the automatic integration of the whole dataset. The changes are sent to the OSM API through your account, grouped by subdivision.

How to contribute

  1. 1

    Sign in with your OSM account

    Use the sign-in button at the bottom of the menu. Your integrations are recorded under your name in the OSM changeset history.

  2. 2

    Pick a brand

    The Brands to integrate page lists every chain for which enrichments are available, with the number of nodes concerned.

  3. 3

    Check the matches

    For every node, check that the ATP ↔ OSM match is right: location, name of the place, tags offered. When in doubt, open the ATP source or the OSM object before accepting.

  4. 4

    Integrate

    If the data looks right, start the integration. The changesets are created automatically on OSM, one per subdivision. The operation cannot be undone: make sure you have checked the data.

Reviewing a brand

The review goes batch by batch: a brand is cut into batches of whole subdivisions, and before integrating a batch you check a sample drawn at random from it. If that sample is right, the data is considered reliable for every place of the batch.

The sample holds at least one example of every tag the batch changes: if the batch adds, say, phone, website, opening_hours and email, the sample will hold a POI carrying each of them. Every tag is therefore seen at least once before the integration, and the sample is filled at random up to 3 items when it holds fewer. Depending on the tags involved, you may well have 4 POIs to check, or more.

Before accepting, make sure the items you checked are representative: try several places in different areas, look at the edge cases (unusual opening hours, badly formatted phone numbers, redirected URLs…). If the slightest doubt remains about the quality of the spider, report an error rather than accept.

Integration statuses

Every integration recorded in the history is given a status reflecting its outcome.

Success

Integration succeeded

Every change has been sent to OpenStreetMap and is now publicly visible.

Partial

Integration partly succeeded

Some subdivisions could not be integrated because of an OpenStreetMap API error. The changes that were sent are visible on OSM all the same. The data left out stays available for a later integration.

Cancelled

Integration cancelled, data reported

The integration was stopped after a contributor found errors in the ATP data. Nothing was sent to OpenStreetMap.

Error

Technical error

An error unrelated to what you did stopped the integration. Nothing was sent to OpenStreetMap. You can start the integration again from the page of the brand concerned after 4 weeks. That delay leaves time to fix the errors the datasets may hold.

Licences

OpenStreetMap

The OSM data is published under the ODbL 1.0 (Open Database License). Every contribution made through this tool falls under the OpenStreetMap Contributor Terms.

All The Places

The ATP data is published under CC0 1.0 (public domain). It can be used freely, without restriction.

atp2osm

The source code of this tool is published under the GPL-3.0 licence. Contributions are welcome on GitHub.

Exporting the data

The data produced by atp2osm can be exported freely, as CSV or JSON, without signing in. It is only the trace of the integrations and the list of the missing brands: the source data stays with the projects that produce it (All The Places and OpenStreetMap).

GET/api/export/history.csv
One row per integration: brand, contributor, date, status, number of objects and of tags.
GET/api/export/subdivisions.csv
The detail per changeset: administrative subdivision, number of objects, OSM changeset identifier and status.
GET/api/export/todo.csv
The brands reported as missing from ATP. The ones ATP now knows are left out, unless ?show_in_atp=1 is passed.
GET/api/stats.json
The figures of the statistics page: totals, series per period, top tags, brands and contributors, spider reliability. Takes the same filters as the page.

Replace .csv with .json for the same content as JSON. The history and todo exports take the same filters as their pages (q, status, user, from, to); the “Export” button above each table carries the active filters over anyway.

Resources