在项目的这一部分,我们首先将数据导入阿里云MaxCompute平台,然后我依托于阿里云MaxCompute平台及PAI平台,对人力资源流动数据进行数据清洗和数据探查。
在MaxCompute中建表代码如下:
DROP TABLE IF EXISTS hr_analytics ;
CREATE TABLE IF NOT EXISTS hr_analytics
(
satisfaction_level DOUBLE
,last_evaluation DOUBLE
,number_project BIGINT
,average_montly_hours BIGINT
,time_spend_company BIGINT
,Work_accident BIGINT
,left1 BIGINT
,promotion_last_5years BIGINT
,position STRING
,salary_level STRING
)
;
然后,进行数据的缺失值处理,检查各个变量的空值数量:
SELECT sum(CASE WHEN satisfaction_level IS NULL THEN 1 ELSE 0 END) AS satisfaction_level
,sum(CASE WHEN last_evaluation IS NULL THEN 1 ELSE 0 END) AS last_evaluation
,sum(CASE WHEN number_project IS NULL THEN 1 ELSE 0 END) AS number_project
,sum(CASE WHEN average_montly_hours IS NULL THEN 1 ELSE 0 END) AS average_montly_hours
,sum(CASE WHEN time_spend_company IS NULL THEN 1 ELSE 0 END) AS time_spend_company
,sum(CASE WHEN work_accident IS NULL THEN 1 ELSE 0 END) AS work_accident
,sum(CASE WHEN left1 IS NULL THEN 1 ELSE 0 END) AS left1
,sum(CASE WHEN promotion_last_5years IS NULL THEN 1 ELSE 0 END) AS promotion_last_5years
,sum(CASE WHEN position IS NULL THEN 1 ELSE 0 END) AS position
,sum(CASE WHEN salary_level IS NULL THEN 1 ELSE 0 END) AS salary_level
FROM hr_analytics
;
检查发现,各项缺失值均为0,即无缺失值存在。
再检查数据的离群值。计算满意度,最近一次评价,参与项目数量,平均工作时间,在职时间,工伤次数的均值及标准差:
SELECT
AVG(satisfaction_level ) AS avg_satisfaction_level
,STDDEV(satisfaction_level ) AS std_satisfaction_level
,AVG(last_evaluation ) AS avg_last_evaluation
,STDDEV(last_evaluation ) AS std_last_evaluation
,AVG(number_project ) AS avg_number_project
,STDDEV(number_project ) AS std_number_project
,AVG(average_montly_hours)AS avg_average_montly_hours
,STDDEV(average_montly_hours ) AS std_average_montly_hours
,AVG(time_spend_company )AS avg_time_spend_company
,STDDEV(time_spend_company ) AS std_time_spend_company
,AVG(work_accident)AS avg_work_accident
,STDDEV(work_accident ) AS std_work_accident
FROM hr_analytics
;
然后利用3σ准则计算离群区间:(μ-3σ,μ+3σ),发现只有time_spend_company 含有离群值,离群值个数为376,离群率为0.4%。根据实际情况,公司员工在职时间的离群数量不多。可能因为突发情况,导致的在时间过长,本实验的目的为预测离职情况,而在职时间过长或过短更有可能影响离职,故不考虑剔除或替换。
然后利用PAI平台进行线性关系检测,得到决定系数R2为0.01,极为接近0,拟合优度很低。再绘制残差图,得到图形如下:
如图所示,图内数值随机分布 说明数据是线性的,即满意度和最后一次评价间有线性关系。