MaPit
=====

pylib/mapit is a standard GeoDjango project; the rest is mostly to fit within
mySociety's standard deployment.

Checking out the code
---------------------

A standard git clone will get you the repository. Some code (used by multiple
mySociety projects) is kept in a submodule repository called commonlib - to get
that, simply type "git submodule update --init" in the main repository
directory and it'll appear.

Installation
------------

Our version is running on Debian lenny - to install PostGIS and GeoDjango I
just followed all the instructions (including creating the template) at
http://docs.djangoproject.com/en/dev/ref/contrib/gis/install/

However, not only is the PostGIS that is installed missing SRID 900913, as
the GeoDjango docs tell you, but both SRID 27700 (British National Grid) and
SRID 29902 (Irish National Grid) are incorrect (and they're quite important
for this application!). After you've installed and got a PostGIS template, log
in to it and update the proj4text column of SRID 27700 to include
+datum=OSGB36, and update SRID 29902 to have +datum=ire65. (This may not be
necessary, depending on your version of PostGIS, but do check.)

Initial instructions for all locations
--------------------------------------

1. Make sure OPTION_STAGING is 0 in conf/general, otherwise you'll run out
   of memory as it tries to remember all the SQL queries made.
2. Optional: turn off escape_string_warning in Postgres' config (unless you
   want your server to run out of disc space logging all the errors, as ours
   did the first time I imported overnight and didn't realise).

United Kingdom
--------------

Here are the basic instructions to install OS OpenData and NSPD Open:

1. Set OPTION_AREA_SRID in conf/general to 27700 (as Boundary-Line shapes are in OSGB).   
2. Download the latest Code-Point Open, Boundary-Line and NSPD from
   <http://parlvid.mysociety.org:81/os/>, and save/unzip in the data directory.
3. Change to the pylib/mapit directory, create database tables and start the site:
   ./manage.py syncdb
   ./manage.py runserver
   # (Alternatively, set up a live server however you wish.)
4. Run the following:
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py import_boundary_line --control=controls.first-gss --commit `ls ../../data/Boundary-Line/*.shp|grep -v high_water`
   # (You can run without --commit to do a dry run.)
   # first-gss in the above assumes the Boundary Line you're importing is
   # October 2010 or later, and uses the new GSS codes.
   ./manage.py find_parents
   ./manage.py import_codepoint ../../data/Code-Point-Open/*.csv
   ./manage.py scilly ../../data/Code-Point-Open/tr.csv
   ./manage.py import_nspd_ni_areas ../../data/NSPD.csv
   ./manage.py import_nspd_ni ../../data/NSPD.csv
   ./manage.py import_nspd_crown_dependencies.py ../../data/NSPD.csv
   ./manage.py generation_activate --commit

For notes on what was done to create generations as you can see on
mapit.mysociety.org, see the end of this file.

Notes on future releases
------------------------

When a new Code-Point is released, you should just be able to run import_codepoint 
and scilly; when new NSPD is out, import_nspd_ni if it's only postcodes that 
have changed, or import_nspd_ni_areas first if boundary changes too (this is 
incomplete, it doesn't use a control file like import_boundary_line does); 
when new Boundary-Line, import_boundary_line and find_parents.

You can manually increase the generation_high_id when something is new and
something else isn't (most commonly, a new Boundary-Line means a new generation
for Great Britain, and you can then increase the Northern Ireland boundaries
manually to be in the new generation).

Norway
------

Here are the basic instructions to install data from OSM:

1. Set OPTION_AREA_SRID in conf/general to 4326 (as OSM shapes are in WGS84).  
2. cd bin and run "python osm_to_kml" to fetch OSM XML and convert it to KML.
3. Change to the pylib/mapit directory, create database tables and start the site:
   ./manage.py syncdb
   ./manage.py runserver
4. Run the following (you can run anything without --commit to do a dry run):
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py import_norway_osm --commit ../../data/cache/*.kml
   ./manage.py generation_activate --commit

Please see below for information on where osm_to_kml gets its OSM data from.

Alternatively, here are the basic instructions to install the N5000 data:

1. Set OPTION_AREA_SRID in conf/general to 4326 (as we'll put N5000 shapes into WGS84).  
2. Download N5000 from
   <http://www.statkart.no/nor/Land/Kart_og_produkter/N5000_-_gratis_oversiktskart/>
   and save/unzip in the data directory.
3. Change to the pylib/mapit directory, create database tables and start the site:
   ./manage.py syncdb
   ./manage.py runserver
4. Run the following (you can run anything without --commit to do a dry run):
   ./manage.py generation_create --commit --desc "Initial import."
   ./manage.py import_norway_n5000 --commit ../../data/N5000\ shape/N5000_AdministrativFlate.shp
   ./manage.py generation_activate --commit

You should now be able to go to /point/4326/10.756389,59.949444 and have
Oslo returned as a result.

Norway OSM data
---------------

The way osm_to_kml works is to fetch a number of fixed and pre-defined
relation IDs from OSM - one (412437) containing all fylke, and then
one for each fylke containing all the kommune inside. These relations
should stay and (now they're correct) not change within OpenStreetmap,
though of course the underlying relations can have their boundaries
improved and so on. See the relation_ids list in the source of
bin/osm_to_kml if you'd like to see the other relation IDs.

The OSM tags 'name', and 'name:no' if 'name' is not set, are used to find
the primary name of the fylke and kommune. In addition, the values of
the tags 'name:smi', 'name:fi' are imported into mapit. Only the
primary name is shown in the mapit web pages and JSON data, while the
other names are stored in the database.

The kommune and fylke number (ID) is fetched from a the tag 'ref' in
OSM, and if it is missing a static list of such IDS in
mapit/data/norway/ids.csv is consulted using the name (for fylke) or
name+fylke (for kommune) as the key.

Improvements / patches
----------------------

Are welcome :)

ATB,
Matthew

=====================================

Notes on creating what's live
-----------------------------

When creating what you see at mapit.mysociety.org, to enable it to have
pre-2010 election boundaries, I ran the above (or rather, what existed at the
time, which is not identical) twice, once with 2009-10 Boundary-Line and then
the 2010-05 edition. I had to write the 2010-05 control file you can see, did
not re-run import_codepoint (as no postcodes had changed), and only ran the NI
stuff the second generation (as we only had current data). The commands I
basically ran are below.

Even worse, as I had to maintain IDs between our old and new versions of mapit,
I then matched up all IDs and names using the scripts in bin, manually inserted
some generation 10 areas (in data) for FixMyStreet and some generation 12 NI
WMC areas for WriteToThem, and manually added our test/fake areas that used to
be in code but can now happily sit in the database along with everything else.
You probably don't need any of that for your own install.

# Create inactive generation.
./manage.py import_boundary_line --control=controls.2009-10 `ls ../../data/Boundary-Line/2009-10/*.shp|grep -v high_water`
./manage.py import_codepoint ../../data/Code-Point-Open-2010-05/*.csv
./manage.py find_parents
# Not importing NI here, as that only has the current boundaries.
./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv
# Make generation active, add another inactive generation
./manage.py import_boundary_line --control=controls.2010-05 `ls ../../data/Boundary-Line/2010-05/*.shp|grep -v high_water`
# import_codepoint not needed as it's the same and there's no P-in-P tests!
./manage.py find_parents
./manage.py scilly ../../data/Code-Point-Open-2010-05/tr.csv # I doubt the boundaries change! But updates the generation.
./manage.py import_nspd ../../data/NSPD-2010-05.csv # This is now split into two scripts, see below.
# Make generation active.