Skip to content

ebwolf/OSM-Tools

Repository files navigation

2024-04-24 Update: Cleaning up, getting working with Python 3.8, and pylinting
2024-04-25 Debugging. Realized 2 things: osm_reader needs a way to better inspect a file.
           Warn the user if it is a multistream bz2 file. Right now program just fails to generate output. 
           Warn the user if missing expected tags (like changesets). Right now program just raises an exception.
           Warn the user about possible memory issues:
               1. Can it run entirely in RAM
               2. Will it need to swap to disk (slow, but will work eventually)
               3. Will it run out of memory entirely.
               These states can be guestimated by checking the amount of physical RAM, total RAM
               And looking at the number of objects to be included. 
               
This is a set of tools specfically designed for working with the OSM Full Planet
file. The tools are designed to work with a fairly stock Python3 install, 
maybe not the version installed with Esri products. I don't have access to those any more.

osm_reader.py - OsmReader is a class for transparently handling the OSM Full Planet which has
    some technical challenges relative to the regular OSM Planet file. Primarily,
    Full Planet History is big. As of 2024-04-24, the file is about 212GB 
    compressed in bz2 format. It also does not have any line breaks.

    OSMReader tranparently handles text .osm, .bz2 and .gz formats.
    it provides two means of traversing the file - tag by tag and object by object

    All other scripts here depend on this class.

osm_chunker.py - Not updated yet, definitely b0rk3d.

osm_fpextract.py -Updating, definitely b0rk3d 
  
     works on smaller files but runs out of memory on big files.
    takes two passes to generate a spatial and/or temporal extract of full planet

osm2sqlite.py - just a placeholder (copy of old very b0rk3d osm_fpextract.py)

osm2fgdb.py - bork3d and likely to remain that way. Does anyone use FileGeoDatabaseses with OSM data?

About

Python scripts for manipulating OSM Full Planet

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages