What happened?
The Machine Learning Scorer accepts several settings it has no way to score. None of them is caught in the editor, so the workflow compiles, runs, and then fails somewhere the message does not point back at the setting that caused it. In one case it does not fail at all.
The first is the Scorer Functions multi-select, which is empty when the operator is first dropped and is not marked required. The operator joins the selected metric names into a Python list literal, so an empty selection comes out as [''] and the metric lookup is asked for the empty name, raising KeyError: ''. This happens on both branches of the Regression switch, since each branch has its own metric list and both start empty.
The second is the pair of column settings. Actual Value and Predicted Value hold one label read twice, so they have to be comparable, but neither the schema nor getOutputSchemas checks that. Picking an integer column against a string column raises Mix of label input types (string and number) from inside scikit-learn for Precision, Recall and F1. With Accuracy the same pair raises nothing at all and scores 0.0, which reads as a model that never predicts correctly rather than as a misconfigured operator. Choosing a non-numeric column while Regression is on fails with could not convert string to float. A column that no longer exists upstream, after a rename or a source change, is not caught either.
Expected: each of these is reported in the editor, against the setting responsible, before the workflow runs.
How to reproduce?
Build a workflow of CSV File Scan feeding a Machine Learning Scorer, over a table holding an integer label column and a string column.
Leave Regression off, set Actual Value and Predicted Value to two columns holding the same labels, and leave Scorer Functions with nothing selected. Run it: the Scorer fails with KeyError: ''.
Now select Accuracy, set Actual Value to the integer column and Predicted Value to the string column, and run it again. The Scorer succeeds and reports an accuracy of 0.0. Switching the metric to Precision Score instead fails with Mix of label input types (string and number).
Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
b59c740
What happened?
The Machine Learning Scorer accepts several settings it has no way to score. None of them is caught in the editor, so the workflow compiles, runs, and then fails somewhere the message does not point back at the setting that caused it. In one case it does not fail at all.
The first is the Scorer Functions multi-select, which is empty when the operator is first dropped and is not marked required. The operator joins the selected metric names into a Python list literal, so an empty selection comes out as
['']and the metric lookup is asked for the empty name, raisingKeyError: ''. This happens on both branches of the Regression switch, since each branch has its own metric list and both start empty.The second is the pair of column settings. Actual Value and Predicted Value hold one label read twice, so they have to be comparable, but neither the schema nor
getOutputSchemaschecks that. Picking an integer column against a string column raisesMix of label input types (string and number)from inside scikit-learn for Precision, Recall and F1. With Accuracy the same pair raises nothing at all and scores 0.0, which reads as a model that never predicts correctly rather than as a misconfigured operator. Choosing a non-numeric column while Regression is on fails withcould not convert string to float. A column that no longer exists upstream, after a rename or a source change, is not caught either.Expected: each of these is reported in the editor, against the setting responsible, before the workflow runs.
How to reproduce?
Build a workflow of CSV File Scan feeding a Machine Learning Scorer, over a table holding an integer label column and a string column.
Leave Regression off, set Actual Value and Predicted Value to two columns holding the same labels, and leave Scorer Functions with nothing selected. Run it: the Scorer fails with
KeyError: ''.Now select Accuracy, set Actual Value to the integer column and Predicted Value to the string column, and run it again. The Scorer succeeds and reports an accuracy of 0.0. Switching the metric to Precision Score instead fails with
Mix of label input types (string and number).Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
b59c740