'GeoMIP', 'HighResMIP', 'ISMIP6', 'LS3MIP', 'LUMIP', 'OMIP', 'PMIP', 'RFMIP', 'ScenarioMIP', 'VolMIP' ] #%% Experiments tmp = [['experiment_id','https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_experiment_id.json'] ] ; experiment_id = readJsonCreateDict(tmp) experiment_id = experiment_id.get('experiment_id') experiment_id = experiment_id.get('experiment_id') ; # Fudge to extract duplicate level # Fix issues keyList = experiment_id.keys() for count,key in enumerate(keyList): entry = experiment_id[key]['parent_activity_id'] print entry #entry = str(entry).replace('[','').replace(']','') entry = entry[0] print entry if isinstance(entry,list) and len(entry) < 2: entry = entry[0] print entry if isinstance(entry,list):
'https://raw.githubusercontent.com/PCMDI/cmip6-cmor-tables/master/Tables/CMIP6_IfxGre.json' ], [ 'Afx', 'https://raw.githubusercontent.com/PCMDI/cmip6-cmor-tables/master/Tables/CMIP6_fx.json' ] ] headerFree = [ 'coordinate', 'frequency', 'formula_terms', 'grid_label', 'nominal_resolution', 'realm', 'region', 'source_id', 'target_mip' ] #%% Loop through tables and create in-memory objects # Loop through tableSource and create output tables tmp = readJsonCreateDict(tableSource) for count, table in enumerate(tmp.keys()): print('table:', table) if table in ['coordinate', 'formula_terms']: vars()[table] = tmp[table] elif table == 'target_mip': vars()[table] = tmp[table].get('activity_id') elif table in headerFree: vars()[table] = tmp[table].get(table) else: vars()[table] = tmp[table] del (tmp, count, table) gc.collect() # Cleanup by extracting only variable lists for count2, table in enumerate(tableSource):
def ascertainVersion(testVal_activity_id, testVal_experiment_id, testVal_frequency, testVal_grid_label, testVal_institution_id, testVal_license, testVal_mip_era, testVal_nominal_resolution, testVal_realm, testVal_required_global_attributes, testVal_source_id, testVal_source_type, testVal_sub_experiment_id, testVal_table_id, commitMessage): # Load current history direct from repo master tmp = [[ 'versionHistory', 'https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/src/versionHistory.json' ]] versionHistory = readJsonCreateDict(tmp) versionHistory = versionHistory.get('versionHistory') versionHistory = versionHistory.get('versionHistory') # Fudge to extract duplicate level del (tmp) versionMIPEra = versionHistory['versions'].get('versionMIPEra') versionCVStructure = versionHistory['versions'].get('versionCVStructure') versionCVContent = versionHistory['versions'].get('versionCVContent') versionCVCommit = versionHistory['versions'].get('versionCVCommit') # Deal with commitMessage formatting commitMessage = commitMessage.replace('\"', '') # versionMIPEra - CMIP6 id - The first integer is “6”, indicating the CV collection is for use in CMIP6 versionMIPEra = versionHistory['versions'].get('versionMIPEra') # versionCVStructure - Incremented when the structure/format of CV’s changes or a new CV is added versionCVStructure = versionHistory['versions'].get('versionCVStructure') # versionCVContent - Incremented when a change to existing content is made other than “source_id” or “institution_id” test1 = [ testVal_activity_id, testVal_experiment_id, testVal_frequency, testVal_grid_label, testVal_license, testVal_mip_era, testVal_nominal_resolution, testVal_realm, testVal_required_global_attributes, testVal_source_type, testVal_sub_experiment_id, testVal_table_id ] test2 = [testVal_institution_id, testVal_source_id] if any(test1): versionCVContent = versionHistory['versions'].get( 'versionCVContent') + 1 versionCVCommit = 0 # Now update versionHistory - can use list entries, as var names aren't locatable if testVal_activity_id: versionHistory['activity_id']['commitMessage'] = commitMessage if testVal_experiment_id: versionHistory['experiment_id']['commitMessage'] = commitMessage if testVal_frequency: versionHistory['frequency']['commitMessage'] = commitMessage if testVal_grid_label: versionHistory['grid_label']['commitMessage'] = commitMessage if testVal_license: versionHistory['license']['commitMessage'] = commitMessage if testVal_mip_era: versionHistory['mip_era']['commitMessage'] = commitMessage if testVal_nominal_resolution: versionHistory['nominal_resolution'][ 'commitMessage'] = commitMessage if testVal_realm: versionHistory['realm']['commitMessage'] = commitMessage if testVal_required_global_attributes: versionHistory['required_global_attributes'][ 'commitMessage'] = commitMessage if testVal_source_type: versionHistory['source_type']['commitMessage'] = commitMessage if testVal_sub_experiment_id: versionHistory['sub_experiment_id'][ 'commitMessage'] = commitMessage if testVal_table_id: versionHistory['table_id']['commitMessage'] = commitMessage # versionCVCommit - Incremented whenever a new source_id and/or institution_id is added or amended elif any(test2): versionCVCommit = versionHistory['versions'].get('versionCVCommit') + 1 # Now update versionHistory - can use list entries, as var names aren't locatable if testVal_institution_id: versionHistory['institution_id']['commitMessage'] = commitMessage if testVal_source_id: versionHistory['source_id']['commitMessage'] = commitMessage # versions - Update versionHistory['versions']['versionMIPEra'] = versionMIPEra versionHistory['versions']['versionCVStructure'] = versionCVStructure versionHistory['versions']['versionCVContent'] = versionCVContent versionHistory['versions']['versionCVCommit'] = versionCVCommit versions = '.'.join( str(x) for x in [versionMIPEra, versionCVStructure, versionCVContent, versionCVCommit]) return [versionHistory, versions]
def ascertainVersion(testVal_activity_id,testVal_experiment_id,testVal_frequency, testVal_grid_label,testVal_institution_id,testVal_license, testVal_mip_era,testVal_nominal_resolution,testVal_realm, testVal_required_global_attributes,testVal_source_id, testVal_source_type,testVal_sub_experiment_id,testVal_table_id, commitMessage): # Load current history direct from repo master tmp = [['versionHistory','https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/src/versionHistory.json'] ] ; versionHistory = readJsonCreateDict(tmp) versionHistory = versionHistory.get('versionHistory') versionHistory = versionHistory.get('versionHistory') ; # Fudge to extract duplicate level del(tmp) versionMIPEra = versionHistory['versions'].get('versionMIPEra') versionCVStructure = versionHistory['versions'].get('versionCVStructure') versionCVContent = versionHistory['versions'].get('versionCVContent') versionCVCommit = versionHistory['versions'].get('versionCVCommit') # Deal with commitMessage formatting commitMessage = commitMessage.replace('\"','') # versionMIPEra - CMIP6 id - The first integer is “6”, indicating the CV collection is for use in CMIP6 versionMIPEra = versionHistory['versions'].get('versionMIPEra') # versionCVStructure - Incremented when the structure/format of CV’s changes or a new CV is added versionCVStructure = versionHistory['versions'].get('versionCVStructure') # versionCVContent - Incremented when a change to existing content is made other than “source_id” or “institution_id” test1 = [testVal_activity_id,testVal_experiment_id,testVal_frequency, testVal_grid_label,testVal_license,testVal_mip_era, testVal_nominal_resolution,testVal_realm, testVal_required_global_attributes,testVal_source_type, testVal_sub_experiment_id,testVal_table_id] test2 = [testVal_institution_id,testVal_source_id] if any(test1): versionCVContent = versionHistory['versions'].get('versionCVContent') + 1 versionCVCommit = 0 # Now update versionHistory - can use list entries, as var names aren't locatable if testVal_activity_id: versionHistory['activity_id']['commitMessage'] = commitMessage if testVal_experiment_id: versionHistory['experiment_id']['commitMessage'] = commitMessage if testVal_frequency: versionHistory['frequency']['commitMessage'] = commitMessage if testVal_grid_label: versionHistory['grid_label']['commitMessage'] = commitMessage if testVal_license: versionHistory['license']['commitMessage'] = commitMessage if testVal_mip_era: versionHistory['mip_era']['commitMessage'] = commitMessage if testVal_nominal_resolution: versionHistory['nominal_resolution']['commitMessage'] = commitMessage if testVal_realm: versionHistory['realm']['commitMessage'] = commitMessage if testVal_required_global_attributes: versionHistory['required_global_attributes']['commitMessage'] = commitMessage if testVal_source_type: versionHistory['source_type']['commitMessage'] = commitMessage if testVal_sub_experiment_id: versionHistory['sub_experiment_id']['commitMessage'] = commitMessage if testVal_table_id: versionHistory['table_id']['commitMessage'] = commitMessage # versionCVCommit - Incremented whenever a new source_id and/or institution_id is added or amended elif any(test2): versionCVCommit = versionHistory['versions'].get('versionCVCommit') + 1 # Now update versionHistory - can use list entries, as var names aren't locatable if testVal_institution_id: versionHistory['institution_id']['commitMessage'] = commitMessage if testVal_source_id: versionHistory['source_id']['commitMessage'] = commitMessage # versions - Update versionHistory['versions']['versionMIPEra'] = versionMIPEra versionHistory['versions']['versionCVStructure'] = versionCVStructure versionHistory['versions']['versionCVContent'] = versionCVContent versionHistory['versions']['versionCVCommit'] = versionCVCommit versions = '.'.join(str(x) for x in [versionMIPEra,versionCVStructure,versionCVContent,versionCVCommit]) return [versionHistory,versions]