OSMSG : A security concerns

Posted by rtnf on 21 January 2023 in English.

So, I just stumbled across this tool, “OpenStreetMap Stats Generator”.

This python tool can automatically download and process current planet file and generate the statistics. Cool.

osmsg [-h] [--start_date START_DATE] [--end_date END_DATE] --username USERNAME --password PASSWORD
             [--timezone {Nepal,UTC}] [--name NAME] [--tags TAGS [TAGS ...]] [--rows ROWS] --url URL [--extract_last_week]
             [--extract_last_day] [--extract_last_month] [--extract_last_year] [--exclude_date_in_name]
             [--format {csv,json,excel,image,text} [{csv,json,excel,image,text} ...]] [--read_from_metadata READ_FROM_METADATA]

But I have some concerns.

Why it needs my OSM username and password? Can I trust this tool? What if this tool store my OSM credentials on 3rd party database or something? Not to mention that, we have to type our password manually in the console, in visible plain text (which is, could be stored too on console’s history).

So I decided to dig the source code to find some clue.

def auth(username, password):
        cookies = verify_me_osm(username, password)
    except Exception as ex:
        raise ValueError("OSM Authentication Failed")

    print("Authenticated !")
    return cookies

Alright, now where the cookies go?

if "geofabrik" in args.url:
        cookies = auth(args.username, args.password)
if "geofabrik" in url:
            cookies_fmt = {}
            test = cookies.split("=")
            # name, value = line.strip().split("=")
            cookies_fmt[test[0]] = f'{test[1]}=="'
            response = requests.get(url, cookies=cookies_fmt)
            response = requests.get(url)

Okay, Geofabric. Does Geofabric really need our OSM credentials or something?

I found some clue here ;

Files which are accessible on our public download server without any login do not contain sensitive data about the OpenStreetMap contributors. The user, uid and changeset fields are missing in these files since May 3, 2018. You can download files with full metadata from a different download server which requires log-in with your OpenStreetMap account. Files from the this non-public download server contain data which is subject to EU data protection regulations. These regulations apply world-wide.

In short, not all Geofabrik planet files require an OSM credential. If we don’t give our OSM credentials, we still can download planet files without personal data of the OSM contributors. But somehow, this python tool forces us to give our OSM credential whenever a geofabrik url is supplied.

Meanwhile, Geofabrik itself provides a safe and proper way to transfer our OSM authorization to 3rd party. That is : (1) redirect to the official page, (2) click “grant access”, (3) redirected back to 3rd party page. This way, 3rd party tool don’t know (and don’t store) our valued OSM username and password.

In short, this python tool might risk your OSM account because it doesn’t implement a security best practices to transfer OSM authorization to 3rd party. Even if this tool doesn’t actually store your OSM password (we might need more investigation regarding this accusation), your password is still visible and recorded to your own console history, in plaintext format.


Comment from TrickyFoxy on 22 January 2023 at 15:50

Have you tried to tell the developer on GitHub about it? He seems to be actively developing the tool at the moment, and he certainly won’t miss the message there.

Comment from rtnf on 22 January 2023 at 16:08


Yes, I already did.

Even though this tool needs some security fixes for more secure public usage, the twitter bot (that used this tool internally by the original developers) is cool though.

Comment from Kshitijraj Sharma on 23 January 2023 at 03:54

Hi @rtnf Thank you for bringing this to my attention. I am actively working on a project related to country level statistics, and initially, my primary source of data was Geofabrik, which required OSM authentication for access to country level changefiles with osm usernames. As I have progressed in the project, I have added support for other planet replication servers. You are correct that, with the exception of Geofabrik, OSM credentials are not required to produce statistics. I will raise an issue to address this concern and work to make the tool more trustworthy. I can confidently say that the tool does not store any credentials and is not connected to the cloud. It is a on-the-fly generator that should only be run on your local terminal. I appreciate your detailed review and thank you for bringing this to my attention.

Comment from Kshitijraj Sharma on 23 January 2023 at 04:12

I have also added an issue to track this concern on the project’s GitHub repository, here, so that the community can follow the progress on this issue. As this is an open-source project, I encourage and welcome contributions from the community.

Login to leave a comment