This program parses the text from the Paizo Pathfinder Reference Document website and turns the data into a series of JSON files. It then can transform those JSON files into an SQLite3 database intended for use with the Pathfinder Open Reference project.
Any third-party libraries that need to be installed are listed in conf/requirements.txt.
You need to ensure you have a src/dir.conf file, that sets up certain environment variables and is sourced by the shell scripts. A template file is provided: src/dir.conf.template.
The functions, under src/psrd/sql, that select data from the database, will not return a list of rows. The cursor will be modified and it is up to the caller to extract rows from the cursor.
The functions, under src/psrd/sql, that insert/update data in the database, will not return the rowid. As above, the cursor will be modified and it is up to the caller to extract that information.
The database's structure is a hierarchy. It uses two forms of hierarchical structure because:
- The two forms have different strengths and weaknesses
- The database is meant to be generated once, and read many times, so the code for writing it can be reasonably centralized.
The first hierarchical structure is the simplest. Each section has a section_id, and every section also has a parent_id. The parent_id references the section_id of the node's parent in the hierarchy. Other then the top node of the hierarchy, all sections have a parent_id.
The second hierarchical structure is detailed in the article by Mike Hillyer entitled Managing Hierarchical Data in MySQL. It is not as simple, but gives a build in way to be able to easily determine all sections that are below a section in the hierarchy, regardless of depth, in one query.
sections
section_id INTEGER PRIMARY KEY, // Primary key
type TEXT NOT NULL, // See TYPES below
subtype TEXT, // See SUBTYPES below
lft INTEGER NOT NULL, // Left side of the section's set, see HIERARCHY
rgt INTEGER NOT NULL, // Right side of the section's set, see HIERARCHY
parent_id INTEGER, // Id of the section's parent, see HIERARCHY
name TEXT, // Section's name. Text
abbrev TEXT, // Section's abbreviation. Text
source TEXT NOT NULL, // Section's source book. Text
description TEXT, // Short description of section's contents. Text
body TEXT); // Section's Body. HTML
ability_types // 1-n with sections
ability_type_id INTEGER PRIMARY KEY, // Primary Key
section_id INTEGER NOT NULL, // Related section
ability_type TEXT); // One of (Extraordinary, Spell-Like, Supernatural)
affliction_details // 1-1 with sections
affliction_details_id INTEGER PRIMARY KEY,// Primary Key
section_id INTEGER NOT NULL, // Related section
contracted TEXT, // How the affliction is contracted. (examples: contact, contact or inhaled, etc) Text
save TEXT, // Save to avoid effect. (example: Fortitude DC 12) Text
onset TEXT, // Time before the affliction impacts the sufferer. (example: 2d4 weeks) Text
frequency TEXT, // Save frequency. (example: 1/round for 2 rounds) Text
effect TEXT, // Effect of the affliction. (example: 1d8 Dex damage) Text
initial_effect TEXT, // Initial effect of the affliction. (example: 1 Con drain) Text
secondary_effect TEXT, // Secondary effect of the affliction. (example: 1 Con damage) Text
cure TEXT // Cure. (example: 1 save). Text
animal_companion_details // 1-1 with sections
animal_companion_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
ac TEXT,
attack TEXT,
ability_scores TEXT,
special_qualities TEXT,
special_attacks TEXT,
size TEXT,
speed TEXT,
level TEXT
class_details // 1-1 with sections
class_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
alignment TEXT, hit_die TEXT);
creature_details // 1-1 with sections
creature_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
sex TEXT,
super_race TEXT,
level TEXT,
cr TEXT,
xp TEXT,
alignment TEXT,
size TEXT,
creature_type TEXT,
creature_subtype TEXT,
init TEXT,
senses TEXT,
aura TEXT,
ac TEXT,
hp TEXT,
fortitude TEXT,
reflex TEXT,
will TEXT,
defensive_abilities TEXT,
dr TEXT,
resist TEXT,
immune TEXT,
sr TEXT,
weaknesses TEXT,
speed TEXT,
melee TEXT,
ranged TEXT,
space TEXT,
reach TEXT,
special_attacks TEXT,
strength TEXT,
dexterity TEXT,
constitution TEXT,
intelligence TEXT,
wisdom TEXT,
charisma TEXT,
base_attack TEXT,
cmb TEXT,
cmd TEXT,
feats TEXT,
skills TEXT,
racial_modifiers TEXT,
languages TEXT,
special_qualities TEXT,
gear TEXT,
environment TEXT,
organization TEXT,
treasure TEXT
database_version
id INTEGER PRIMARY KEY, // Primary Key
version INTEGER // Database's schema version
feat_type_descriptions // 1-1 with sections
feat_type_id INTEGER PRIMARY KEY, // Primary Key
section_id INTEGER NOT NULL, // Related section
feat_type_description TEXT // Stores the complete list as one text string of the feat types. Needed for performance.
feat_types // 1-n with sections
feat_type_id INTEGER PRIMARY KEY, // Primary Key
section_id INTEGER NOT NULL, // Related section
feat_type TEXT // Stores one feat type. Used for creating feat lists by type.
item_details // 1-1 with sections
item_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
slot TEXT,
cl TEXT,
price TEXT,
weight TEXT,
requirements TEXT,
skill TEXT,
cr_increase TEXT,
cost TEXT
settlement_details // 1-1 with sections
settlement_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
alignment TEXT,
settlement_type TEXT,
size TEXT,
corruption TEXT,
crime TEXT,
economy TEXT,
law TEXT,
lore TEXT,
society TEXT,
qualities TEXT,
danger TEXT,
disadvantages TEXT,
government TEXT,
population TEXT,
base_value TEXT,
purchase_limit TEXT,
spellcasting TEXT,
minor_items TEXT,
medium_items TEXT,
major_items TEXT
skill_attributes // 1-1 with sections
skill_attribute_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
attribute TEXT,
armor_check_penalty INT,
trained_only INT
spell_components // 1-n with sections
spell_component_id INTEGER PRIMARY KEY,
section_id INTEGER NO NULL,
component_type TEXT NOT NULL,
description TEXT,
notable INT NOT NULL
spell_descriptors // 1-n with sections
spell_descriptor_id INTEGER PRIMARY KEY,
section_id INTEGER NO NULL,
descriptor TEXT NOT NULL
spell_details // 1-1 with sections
spell_detail_id INTEGER PRIMARY KEY,
section_id INTEGER NO NULL,
school TEXT NOT NULL,
subschool TEXT,
descriptor_text TEXT,
level_text TEXT,
casting_time TEXT,
preparation_time TEXT,
range TEXT,
duration TEXT,
saving_throw TEXT,
spell_resistance TEXT,
as_spell_id INT
spell_effects // 1-n with sections
spell_effect_id INTEGER PRIMARY KEY,
section_id INTEGER NO NULL,
name TEXT NOT NULL,
description TEXT NOT NULL
spell_lists // 1-n with sections
spell_list_id INTEGER PRIMARY KEY,
section_id INTEGER NO NULL,
level INTEGER NOT NULL,
class TEXT,
magic_type TEXT
trap_details // 1-1 with sections
trap_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
trap_type TEXT,
perception TEXT,
disable_device TEXT,
duration TEXT,
effect TEXT,
trigger TEXT,
reset TEXT
vehicle_details // 1-1 with sections
vehicle_details_id INTEGER PRIMARY KEY,
section_id INTEGER NOT NULL,
size TEXT,
vehicle_type TEXT,
squares TEXT,
cost TEXT,
ac TEXT,
hardness TEXT,
hp TEXT,
base_save TEXT,
maximum_speed TEXT,
acceleration TEXT,
cmb TEXT,
cmd TEXT,
ramming_damage TEXT,
propulsion TEXT,
driving_check TEXT,
forward_facing TEXT,
driving_device TEXT,
driving_space TEXT,
decks TEXT,
deck TEXT,
weapons TEXT,
crew TEXT,
passengers TEXT
ability
- description: Contains any section that represents an ability. Many subtypes are shared with section. For example, rogue talents sometimes grant abilities, but not always.
- subtypes: barbarian_rage_power, bardic_performance, gunslinger_deed, magus_arcana, ninja_trick, rogue_advanced_talent, rogue_talent, summoner_evolution_1, summoner_evolution_2, summoner_evolution_3, summoner_evolution_4, witch_grand_hex, witch_hex, witch_major_hex
- tables: ability_types
affliction
- description:
- subtypes: addiction, curse, disease, infestation, insanity, poison
- tables: affliction_details
animal_companion
- description: Section describes an animal companion
- subtypes: None
- tables: animal_companion_details
class
- description: Section describes a class
- subtypes: base, core, npc, prestige
- tables: class_details
class_archetype
- description: Section describes a class archetype
- subtypes: barbarian, bard, druid, fighter, monk, paladin, ranger, rogue
- tables: None
creature
- description: Section describes a creature, monster or npc
- subtypes: familiar
- tables: creature_details
feat
- description: Section describes a feat
- subtypes: None
- tables: feat_type_descriptions, feat_types
haunt
- description: Section describes a haunt (Game Mastery Guide)
- subtypes: None
- tables: None
item
- description: Section describes an item
- subtypes: armor, arms, belt, body, chest, eyes, feet, hands, head, headband, neck, ring, shield, shoulders, wrist
- tables: item_details
list
- description: A top level section made to hold all sections of a certain type (Feats, Spells, Rules from Ultimate Magic, etc
- subtypes: None
- tables: None
race
- description: Section describes a race
- subtypes: monster_race, standard_race
- tables: None
racial_trait
- description: Section describes a racial trait
- subtypes: dwarves, elves, gnomes, half-elves, half-orcs, halflings, humans
- tables: None
section
- description: Contains a section of text. Subtype may indicate a special section subtype.
- subtypes: alchemist_discovery, arcane_school, barbarian_rage_power, cleric_domain, cleric_subdomain, condition, elemental_arcane_school, focused_arcane_school, ninja_trick, oracle_mystery, ranger_combat_style, rogue_advanced_talent, rogue_talent, sorcerer_bloodline, special_abilities, spellbook, warrior_order, witch_patron
- tables: None
settlement
- description: Section describes a settlement (Game Mastery Guide)
- subtypes: None
- tables: settlement_details
skill
- description: Section describes a skill
- subtypes: None
- tables: skill_attributes
spell
- description: Section describes a spell
- subtypes: None
- tables: spell_components, spell_descriptors, spell_details, spell_effects, spell_lists
table
- description: Section represents a table from the text.
- subtypes: None
- tables: None
- notes: Later the table should be broken down into rows to give access to the contained information.
trap
- description: Section describes a trap
- subtypes: None
- tables: trap_details
vehicle
- description: Section describes a vehicle (Ultimate Combat)
- subtypes: None
- tables: vehicle_details