round_to_precision method destroys validity

[Adam-Mazur]

Hello, I discovered a bug with the Dice implementation that flips the predictions of the classifier, making the counterfactuals invalid.

I'm using the diabetes dataset, with features scaled to 0 and 1, and a LGBM classifier. I have 90 instances from the dataset for which I'm generating the counterfactuals using Dice with:

• model backend: sklearn
• model func: None
• method: genetic
• proximity_weight: 0.2
• diversity_weight: 0.05
• sparsity_weight: 0.05
• stopping_threshold: 0.5
• total_CFs: 5

I'm calculating the average validity of found counterfactuals with the following code:

def calculate_validity(
    model: BaseClassifier, counterfactuals: np.ndarray, target_classes: np.ndarray
) -> np.ndarray:
    validity = np.zeros(counterfactuals.shape[0])
    for i, (cf, target_class) in enumerate(zip(counterfactuals, target_classes)):
        if model.predict_crisp(cf)[0] == target_class:
            validity[i] = 1
    return validity

When running Dice with the above config I was getting the following metrics:

diversity: 0.378 ± 0.009
proximity_l1: 0.787 ± 0.013
proximity_l2: 0.399 ± 0.007
validity: 0.909 ± 0.014
plausibility: 0.252 ± 0.004

As you can see, the validity was around 90%, which was weird to me, because I though Dice was filtering invalid counterfactuals before returning the results. I looked at the code and discovered that if I change this code (dice_genetic.py) from:

        if self.final_cfs_df is not None:
            self.final_cfs_df[self.data_interface.outcome_name] = self.cfs_preds
            self.final_cfs_df_sparse[self.data_interface.outcome_name] = self.cfs_preds
            self.round_to_precision()

to:

        if self.final_cfs_df is not None:
            self.final_cfs_df[self.data_interface.outcome_name] = self.cfs_preds
            self.final_cfs_df_sparse[self.data_interface.outcome_name] = self.cfs_preds

removing the round_to_precision method the metrics would change to:

diversity: 0.377 ± 0.009 (count=90)
proximity_l1: 0.784 ± 0.013 (count=450)
proximity_l2: 0.399 ± 0.007 (count=450)
validity: 1.000 ± 0.000 (count=450)
plausibility: 0.251 ± 0.004 (count=450)