# %% [markdown] # To demonstrate the performance of the meta-model, we can create an # alternate design of experiments. Note that to get different random values, # we set the `random_seed` argument to something other than the default value. # %% design2 = design_experiments( scope=road_scope, db=emat_db, n_samples_per_factor=10, sampler='lhs', random_seed=2, ) # %% design2_results = mm.run_experiments(design2) design2_results.head() # %% mm.cross_val_scores() # %% [markdown] # ### Compare Core vs Meta Model Results # # We can generate a variety of plots to compare the distribution of meta-model outcomes # on the new design against the original model's results. # %% from emat.analysis import contrast_experiments contrast_experiments(road_scope, lhs_results, design2_results)
# %% [markdown] # ## Contrasting Sets of Experiments # # A similar set of visualizations can be created to contrast two set # of experiments derived from the same (or substantially similar) scopes. # This is particularly valuable to evaluate the performance of meta-models # that are derived from core models, as we can generate scatter plot # matrices that show experiments from both the core and meta models. # # To demonstrate this capability, we'll first create a meta-model from # the Road Test core model, then apply that meta-model to a design of # 5,000 experiments to create a set of meta-model results to visualize. # %% mm = model.create_metamodel_from_design('lhs') mm_design = mm.design_experiments(n_samples=5000) mm_results = mm.run_experiments(mm_design) # %% [markdown] # The `contrast_experiments` function in the `emat.analysis` sub-package can # automatically create a scatter plot matrix, using a very similar interface # to the `display_experiments` function. The primary difference between these # two functions is that `contrast_experiments` takes two sets of experiments # as arguments, instead of one. The resulting plots are also not colorized # based on the roles of each factor in the scope; instead colors are used # to differentiate the different datasets. # %% from emat.analysis import contrast_experiments contrast_experiments(scope, mm_results, results)
# %% [markdown] # We can then create a meta-model automatically from these experiments. # %% mm = model.create_metamodel_from_design('lhs') mm # %% [markdown] # If you are using the default meta-model regressor, as we are doing here, # you can directly access a cross-validation method that uses the experimental # data to evaluate the quality of the regression model. The `cross_val_scores` # provides a measure of how well the meta-model predicts the experimental # outcomes, similar to an $R^2$ measure on a linear regression model. # %% mm.cross_val_scores() # %% [markdown] # We can apply the meta-model directly on a new design of experiments, and # use the `contrast_experiments` visualization tool to review how well the # meta-model is replicating the underlying model's results. # %% design2 = mm.design_experiments(design_name='lhs_meta', n_samples=5000) results2 = mm.run_experiments(design2) # %% from emat.analysis import contrast_experiments contrast_experiments(mm.scope, results2, results)