Additional Training Information

Customization of additional metrics, parameters, and tags against run_id in the current experiment after saving the model.

Note

  • You can only use these api once the model is saved either in Stuio using Watson Machine Learning or as external models in Catalog.

  • Once the model is available, set the model using model=facts_client.assets.get_model() and you can use model.get_experiment() so all experiment management utilities APIs will be available to use.

Attention

Supported only for predictive models, not for prompt template asset

Experiment management utilities

class NotebookExperimentUtilities

Bases: object

Model notebook experiment utilities. Running client.assets.model() makes all methods in NotebookExperimentUtilities object available to use.

get_info()

Get experiment info. Supported for CPD version >=4.6.4

A way to use me is:

>>> exp=model.get_experiment()
>>> exp.get_info()
get_run(run_id: str = None) → NotebookExperimentRunsUtilities

Get run object available in notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:

>>> run=exp.get_run() # returns latest run object available
>>> run=exp.get_run(run_id=run_id) # you can get specific run object
get_all_runs()

Get all runs available in notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:

>>> exp.get_all_runs()
get_experiment(self, experiment_name: str = None) → NotebookExperimentUtilities

Get notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:

>>> exp=model.get_experiment() # returns  experiment
>>> exp=model.get_experiment(experiment_name="test") # In case notebook experiment not available, you can initiate a new one and run to add details.

Runs Management Utilities

Set Operations

set_custom_metric(self, metric_id: str, value: float) → None

Set model training metric against run_id

Parameters:

  • metric_id (str) – Metric key name

  • value (float) – Metric value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:

>>> run.set_custom_metric(metric_key=<key>,value=<value>)
set_custom_param(self, param_id: str, value: str) → None

Set model training param against the run_id

Parameters:

  • param_id (str) – Param key name

  • value (str) – Param value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:

>>> run.set_custom_param(param_id=<key>,value=<value>)
set_custom_tag(self, tag_id: str, value: str) → None

Set model training tag against the run_id

Parameters:

  • tag_id (str) – Tag key name

  • value (str) – Tag value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:

>>> run.set_custom_tag(tag_id=<key>,value=<value>)
set_custom_metrics(self, metrics_dict: Dict[str, Any], debug: bool = False) → None

Set model training metrics against the run_id.

Parameters:

  • metrics_dict (dict) – Metric key, value pairs.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

Raises:

ClientError – Raises client error for exceptions.

Returns:

None

A way to use me is:

>>> run.set_custom_metrics(metrics_dict={"training_score": 0.955, "training_mse": 0.911})
set_custom_params(self, params_dict: Dict[str, Any], debug: bool = False) → None

Set model training params against the run_id.

Parameters:

  • params_dict (dict) – Params key, value pairs.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

Raises:

ClientError – Raises client error for exceptions.

Returns:

None

A way to use me is:

>>> run.set_custom_params(params_dict={"num_class": 3, "early_stopping_rounds": 10})
set_custom_tags(self, tags_dict: Dict[str, Any], debug: bool = False) → None

Set model training tags against the run_id.

Parameters:

  • tags_dict (dict) – Tags key, value pairs.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

Raises:

ClientError – Raises client error for exceptions.

Returns:

None

A way to use me is:

>>> run.set_custom_tags(tags_dict={"tag1": <value>, "tag2": <value>})
set_custom_run_facts(self, metrics: Dict[str, Any] = None, params: Dict[str, Any] = None, tags: Dict[str, Any] = None, overwrite: bool = False, debug: bool = False)

Set custom run facts for an experiment.

Parameters:

  • metrics (dict, optional) – A dictionary containing metrics.

  • params (dict, optional) – A dictionary containing parameters.

  • tags (dict, optional) – A dictionary containing tags.

  • overwrite (bool, optional) – Whether to overwrite existing facts for a selected run. Default is False.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

A way to use me is:

>>> metrics = {"training_score": 0.955, "training_mse": 0.911}
    params = {"batch_size": 32, "learning_rate": 0.001}
    tags = {"experiment_type": "CNN", "dataset": "MNIST"}
>>> run.set_custom_run_facts(metrics_dict=metrics, params_dict=params, tags_dict=tags, overwrite=False, debug=True)

Remove Operations

remove_custom_metric(self, metric_id: str) → None

Remove metric by id against the run_id

Parameters:

metric_id (str) – Metric id to remove.

A way you might use me is:

>>> run.remove_custom_metric(metric_id=<metric_id>)
remove_custom_param(self, param_id: str) → None

Remove param by id against the run_id

Parameters:

param_id (str) – Param id to remove.

A way you might use me is:

>>> run.remove_custom_param(param_id=<param_id>)
remove_custom_tag(self, tag_id: str) → None

Remove tag by id against the run_id

Parameters:

tag_id (str) – Tag id to remove.

A way you might use me is:

>>> run.remove_custom_tag(tag_id=<tag_id>)
remove_custom_metrics(self, metric_ids: List[str]) → None

Remove multiple metrics.

Parameters:

metric_ids (list of str) – List of metric ids to remove from run.

A way you might use me is:

>>> model.remove_custom_metrics(metric_ids=["id1", "id2"])
remove_custom_params(self, param_ids: List[str], debug: bool = False) → None

Remove multiple params against the run_id.

Parameters:

  • param_ids (list of str) – List of param ids to remove from run.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

A way you might use me is:

>>> run.remove_custom_params(param_ids=["id1", "id2"])
remove_custom_tags(self, tag_ids: List[str], debug: bool = False) → None

Remove multiple tags against the run_id.

Parameters:

  • tag_ids (list of str) – List of tag ids to remove from run.

  • debug (bool, optional) – Whether to display additional details for debugging purposes. Default is False.

A way you might use me is:

>>> run.remove_custom_tags(tag_ids=["id1", "id2"])
remove_custom_run_facts(self, metric_ids: List[str] = None, param_ids: List[str] = None, tag_ids: List[str] = None, debug: bool = False) → None

Remove multiple custom run facts (metrics,params or tags) from the run.

Parameters:

  • tag_ids – List of tag IDs to remove.

  • param_ids – List of parameter IDs to remove.

  • metric_ids – List of metric IDs to remove.

  • optional) (debug (bool,) – Used to display additional details for debugging purpose. Default is False.

At least one of tag_ids, param_ids, or metric_ids must be provided.

Example usage:

>>> run.remove_custom_run_facts(metric_ids=["metric1"], param_ids=["param1"],tag_ids=["tag1", "tag2"])