model type Date Filename Notes CV Performance
LightGBM 12/17 policy_1217_1_fd1,2,3.txt hyper params used: 1217. unsegmented data; all features; NO weighted training n/a
12/20 policy_1220_1_fd1,2,3.txt hyper params used: 1217. unsegmented data; all features; VANILLA weighted training n/a
12/23 policy_1223_m_7d_seg_200_fd1,2,3.txt hyper params used: 1217. segmented data; feature set: only generosity ratio; vanilla weighted training fd1: recall: 0.701 fd2: recall: 0.315 fd3: recall: 1
01/03 policy_0103_m_7d_seg_200_fd1,2,3.txt hyper params used: 1217. segmented data; feature set: only generosity ratio; NO weighted training fd1: recall: 0.0 fd2: recall: 0.168 fd3: recall: 0.103
policy_0103_f_7d_seg_200_fd1,2,3.pkl hyper params used: 1217. segment 200; feature set: generosity ratio, freq_visit_7d, freq_purchase_7d; vanilla weighted training fd1: recall: 1 fd2: recall: 1 fd3: recall: 1
policy_0103_f_14d_seg_200_fd1,2,3.pkl hyper params used: 1217. segment 200; feature set: generosity ratio, freq_visit_14d, freq_purchase_14d; vanilla weighted training fd1: recall: 1 fd2: recall: 1 fd3: recall: 1
policy_0103_f_30d_seg_200_fd1,2,3.pkl hyper params used: 1217. segment 200; feature set: generosity ratio, freq_visit_30d, freq_purchase_30d; vanilla weighted training fd1: recall: 1 fd2: recall: 1 fd3: recall: 1
policy_0103_f_7d_seg_110_fd1,2,3.pkl hyper params used: 1217. segment 110; feature set: generosity ratio, freq_visit_7d, freq_purchase_7d; vanilla weighted training recall & true neg rate: fd1: 0.99/0.99, fd2: 0.997/0.989, fd3: 0.996/0.992
policy_0103_f_7d_seg_000_fd1,2,3.pkl hyper params used: 1217. segment 000; feature set: generosity ratio, freq_visit_7d, freq_purchase_7d; vanilla weighted training recall & true neg rate: fd1: 0.966/0.968 fd2: 0.979/0.966, fd3: 0.942/0.969
01/05 policy_0105_ef_7d_fd1,2,3.pkl hyper params used: 1217. NO segment; feature set: end_bf_receive_marker, generosity ratio, freq_visit_7d, freq_purchase_7d; vanilla weighted training recall & true neg rate: fd1: 1/1 fd2: 1/1, fd3: 1/1
01/06 ROI_0106_m_7d_seg_001_fd1,2,3.pkl no invalid coupons in train/val sets in CV—————————-hyper params used: 1217. see doc for the corresponding “m_7d” features. vanilla weighted training recall & true neg rate: fd1: 0.736/0.926 fd2: 0.759/0.919, fd3: 0.714/0.909
ROI_0106_m_7d_seg_010_fd1,2,3.pkl recall & true neg rate: fd1: 0.655/0.814 fd2: 0.805/0.732, fd3: 0.8/0.817
ROI_0106_m_7d_seg_000_fd1,2,3.pkl recall & true neg rate: fd1: 0.839/0.7 fd2: 0.841/0.731, fd3: 0.873/0.665
ROI_0106_m_7d_seg_011_and_021_fd1,2,3.pkl recall & true neg rate: fd1: 0.984/0.864 fd2: 0.984/0.913, fd3: 0.985/0.844
ROI_0106_m_7d_oth_segs_fd1,2,3.pkl recall & true neg rate: fd1: 0.77/0.693 fd2: 0.727/0.706, fd3: 0.767/0.702
1/10 ROI_0110_m_7d_seg_001.pkl, ROI_0110_m_7d_seg_011_and_021.pkl, ROI_0110_m_7d_seg_010.pkl, ROI_0110_m_7d_seg_000.pkl, ROI_0110_m_7d_oth_segs.pkl no invalid coupons in ROI train/test sets—————————hyper params used: 1217. see doc for the corresponding “m_7d” features. vanilla weighted training. —————————— Recall & true neg rate for each segments see doc in experiment_results/perf_ROI_0110_m_7d.txt. ——————————Overall perf: 0.79(recall)/0.68(trueNeg)—————————— Business metrics: - the redemption rate of top5% raw-prediction score coupons: 22.9% - the net profit each top5% raw-prediction score coupon brings in: ￥5.79 - the net profit of each predicted-as-redeemed coupon brings in at the 0.5 decision threshold: ￥2.68 (Compared to the ￥1.32 each coupon receipt brings in originally) n/a
CatBoost 1/15 hyper params: 0115; feature set m_7d; weighted training———————————overall perf: 0.768(Recall)/0.667(trueNeg)———————————Business metrics: - the redemption rate of top5% raw-prediction score coupons: 18.9% - the net profit each top5% raw-prediction score coupon brings in: ￥4.87 - the net profit of each predicted-as-redeemed coupon brings in at the 0.5 decision threshold: ￥2.51 (Compared to the ￥1.32 each coupon receipt brings in originally)