fader network training failure

[413029811]

I used exactly the same params as yours in README.md to train the fader network to manipulate "EyeGlasses" attribute. But the clf_accu_Eyeglasses_1 always decreased to a low value at around 750 epoch, like this:
INFO - 10/18/18 12:58:35 - 1:41:47 - 048928 - Latent discriminator : 0.11062 / Reconstruction loss : 0.00315
INFO - 10/18/18 12:58:39 - 1:41:52 - 049728 - Latent discriminator : 0.05209 / Reconstruction loss : 0.00319
INFO - 10/18/18 12:58:41 - 1:41:53 -
INFO - 10/18/18 12:59:20 - 1:42:33 - Latent discriminator accuracy:
INFO - 10/18/18 12:59:20 - 1:42:33 - lat_dis_accu : 77.984%
INFO - 10/18/18 12:59:20 - 1:42:33 - lat_dis_accu_Eyeglasses: 77.984%
INFO - 10/18/18 12:59:20 - 1:42:33 -
INFO - 10/18/18 13:00:25 - 1:43:38 - Classifier accuracy:
INFO - 10/18/18 13:00:25 - 1:43:38 - clf_accu : 50.873%
INFO - 10/18/18 13:00:25 - 1:43:38 - clf_accu_Eyeglasses : 50.873%
INFO - 10/18/18 13:00:25 - 1:43:38 - clf_accu_Eyeglasses_0: 94.946%
INFO - 10/18/18 13:00:25 - 1:43:38 - clf_accu_Eyeglasses_1: 6.800%
INFO - 10/18/18 13:00:25 - 1:43:38 -
INFO - 10/18/18 13:00:25 - 1:43:38 - Autoencoder loss: 0.00326

And the best evaluation accuracy ever riched is :
INFO - 10/17/18 21:27:23 - 3 days, 4:35:00 - Latent discriminator accuracy:
INFO - 10/17/18 21:27:23 - 3 days, 4:35:00 - lat_dis_accu : 93.114%
INFO - 10/17/18 21:27:23 - 3 days, 4:35:00 - lat_dis_accu_Eyeglasses: 93.114%
INFO - 10/17/18 21:27:23 - 3 days, 4:35:00 -
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - Classifier accuracy:
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - clf_accu : 79.828%
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - clf_accu_Eyeglasses : 79.828%
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - clf_accu_Eyeglasses_0: 98.878%
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - clf_accu_Eyeglasses_1: 60.779%
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 -
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - Autoencoder loss: 0.00338
INFO - 10/17/18 21:28:28 - 3 days, 4:36:05 - Best evaluation accuracy: 0.79828

here is the params I used:
--img_sz 256
--img_fm 3
--attr "Eyeglasses"

--instance_norm False
--init_fm 32
--max_fm 512
--n_layers 6
--n_skip 0
--deconv_method "convtranspose"
--hid_dim 512
--dec_dropout 0
--lat_dis_dropout 0.3

--n_lat_dis 1
--n_ptc_dis 0
--n_clf_dis 0
--smooth_label 0.2
--lambda_ae 1
--lambda_lat_dis 0.0001
--lambda_ptc_dis 0
--lambda_clf_dis 0
--lambda_schedule 500000
--v_flip False
--h_flip True
--batch_size 32
--ae_optimizer "adam,lr=0.0002"
--dis_optimizer "adam,lr=0.0002"
--clip_grad_norm 5
--n_epochs 1000
--epoch_size 50000

--ae_reload ""
--lat_dis_reload ""
--ptc_dis_reload ""
--clf_dis_reload ""
--eval_clf "models/default/kjrite0bvw/best.pth"
--debug False

and the pytorch version is 0.4.1. Any idea why this happened? Thanks.