Additional Training Information

Customization of additional metrics, parameters, and tags against run_id in the current experiment after saving the model.

Note

You can only use these api once the model is saved either in Stuio using Watson Machine Learning or as external models in Catalog.
Once the model is available, set the model using model=facts_client.assets.get_model() and you can use model.get_experiment() so all experiment management utilities APIs will be available to use.

Attention

Supported only for predictive models, not for prompt template asset

Experiment management utilities

class NotebookExperimentUtilities
Bases: object

Model notebook experiment utilities. Running client.assets.model() makes all methods in NotebookExperimentUtilities object available to use.
get_info()
Get experiment info. Supported for CPD version >=4.6.4

A way to use me is:
>>> exp=model.get_experiment()
>>> exp.get_info()
get_run(run_id: str = None) → NotebookExperimentRunsUtilities
Get run object available in notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:
>>> run=exp.get_run() # returns latest run object available
>>> run=exp.get_run(run_id=run_id) # you can get specific run object
get_all_runs()
Get all runs available in notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:
>>> exp.get_all_runs()
get_experiment(self, experiment_name: str = None) → NotebookExperimentUtilities
Get notebook experiment. Supported for CPD version >=4.6.4

A way to use me is:
>>> exp=model.get_experiment() # returns  experiment
>>> exp=model.get_experiment(experiment_name="test") # In case notebook experiment not available, you can initiate a new one and run to add details.

Runs management utilities

class NotebookExperimentRunsUtilities
Bases: object

Model notebook experiment runs utilities. Running client.assets.model() makes all methods in NotebookExperimentUtilities object available to use.
get_info()
Get run info. Supported for CPD version >=4.6.4

A way to use me is:
>>> run=exp.get_run() # returns latest if run_id is not given
>>> run.get_info()
set_custom_metric(metric_id: str, value: float) → None
Set model training metric

Parameters:

metric_id (str) – Metric key name

value (float) – Metric value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_metric(metric_key=<key>,value=<value>)
set_custom_metrics(metrics_dict: Dict[str, Any]) → None
Set model training metrics

Parameters:

metrics_dict (dict) – Metric key,value pairs.

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_metrics(metrics_dict={"training_score":0.955, "training_mse":0.911})
set_custom_param(param_id: str, value: str) → None
Set model training param

Parameters:

param_id (str) – Param key name

value (str) – Param value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_param(param_id=<key>,value=<value>)
set_custom_params(params_dict: Dict[str, Any]) → None
Set model training params

Parameters:

params_dict (dict) – Params key,value pairs.

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_params(params_dict={"num_class":3,"early_stopping_rounds":10})
set_custom_tag(tag_id: str, value: str) → None
Set model training tag

Parameters:

tag_id (str) – Tag key name

value (str) – Tag value

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_tag(tag_id=<key>,value=<value>)
set_custom_tags(tags_dict: Dict[str, Any]) → None
Set model training tags

Parameters:

tags_dict (dict) – Tags key,value pairs.

Raises:

ClientError – Raises client error for exceptions

Returns:

None

A way to use me is:
>>> model.set_custom_tags(tags_dict={"tag1":<value>,"tag2":<value>})
get_custom_metric(metric_id: str) → List
Get custom metric value by id

Parameters:

metric_id (str) – Custom metric id to retrieve.

Return type:

list

A way you might use me is:
>>> model.get_custom_metric_by_id(metric_id="<metric_id>")
get_custom_metrics(metric_ids: List[str] = None) → List
Get all logged custom metrics

Parameters:

metrics_ids (list) – (Optional) Metrics ids to get. If not provided, returns all metrics available for the latest run

Return type:

list[dict]

A way you might use me is:
>>> model.get_custom_metrics() # uses last logged run
>>> model.get_custom_metrics(metric_ids=["id1","id2"]) # uses last logged run
get_custom_param(param_id: str) → Dict
Get custom param value by id

Parameters:

param_id (str) – Custom param id to retrieve.

Return type:

list

A way you might use me is:
>>> model.get_custom_param(param_id="<param_id>")
get_custom_params(param_ids: List[str] = None) → List
Get all logged params

Parameters:

param_ids (list) – (Optional) Params ids to get. If not provided, returns all params available for the latest run

Return type:

list[dict]

A way you might use me is:
>>> model.get_custom_params() # uses last logged run
>>> model.get_custom_params(param_ids=["id1","id2"]) # uses last logged run
get_custom_tag(tag_id: str) → Dict
Get custom tag value by id

Parameters:

tag_id (str) – Custom tag id to retrieve.

Return type:

dict

A way you might use me is:
>>> model.get_custom_tag(tag_id="<tag_id>")
get_custom_tags(tag_ids: List[str] = None) → List
Get all logged tags

Parameters:

tag_ids (list) – (Optional) Tags ids to get. If not provided, returns all tags available for the latest run

Return type:

list[dict]

A way you might use me is:
>>> model.get_custom_tags() # uses last logged run
>>> model.get_custom_tags(tag_ids=["id1","id2"]) # uses last logged run
remove_custom_metric(metric_id: str) → None
Remove metric by id

Parameters:

metric_id (str) – Metric id to remove.

A way you might use me is:
>>> model.remove_custom_metric(metric_id=<metric_id>)
remove_custom_metrics(metric_ids: List[str]) → None
Remove multiple metrics

Parameters:

metric_ids (list) – Metric ids to remove from run.

A way you might use me is:
>>> model.remove_custom_metrics(metric_ids=["id1","id2"]) #uses last logged run
remove_custom_param(param_id: str) → None
Remove param by id

Parameters:

param_id (str) – Param id to remove.

A way you might use me is:
>>> model.remove_custom_param(param_id=<param_id>)
remove_custom_params(param_ids: List[str]) → None
Remove multiple params

Parameters:

param_ids (list) – Param ids to remove from run.

A way you might use me is:
>>> model.remove_custom_params(param_ids=["id1","id2"])
remove_custom_tag(tag_id: str) → None
Remove tag by id

Parameters:

tag_id (str) – Tag id to remove.

A way you might use me is:
>>> model.remove_custom_tag(tag_id=<tag_id>)
remove_custom_tags(tag_ids: List[str]) → None
Remove multiple tags

Parameters:

tag_ids (list) – Tag ids to remove from run.

A way you might use me is:
>>> model.remove_custom_tags(tag_ids=["id1","id2"])