一、现象
最近发现Mysql服务器磁盘IO一直很高
[root@push-5-221 ~]# iostat -k -d -x 1 10 Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.08 0.00 14.89 0.09 177.79 23.88 0.34 23.05 5.42 23.05 11.01 16.40 dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.49 0.07 380.21 5.59 387.46 12.69 0.24 dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.98 0.00 91.25 12.04 92.94 6.55 0.00 dm-2 0.00 0.00 0.00 14.79 0.00 174.15 23.56 0.33 22.27 8.84 22.27 10.96 16.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 83.00 0.00 650.50 15.67 2.55 30.66 0.00 30.66 12.02 99.80 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 83.00 0.00 636.50 15.34 2.55 30.66 0.00 30.66 12.02 99.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 88.00 0.00 684.50 15.56 2.60 29.42 0.00 29.42 11.35 99.90 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 87.00 0.00 677.00 15.56 2.60 29.76 0.00 29.76 11.49 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 87.00 0.00 687.50 15.80 2.48 28.77 0.00 28.77 11.51 100.10 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 88.00 0.00 699.00 15.89 2.48 28.48 0.00 28.48 11.36 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 88.00 0.00 756.50 17.19 2.48 28.22 0.00 28.22 11.35 99.90 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 86.00 0.00 740.00 17.21 2.48 28.87 0.00 28.87 11.62 99.90
%util已经接近100%了,IO性能严重瓶颈,这台服务器有两个Mysql实例,3306是主库,3307是另一台Mysql的从库,进入从库查看从库状态
mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 172.28.5.223 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-223.000012 Read_Master_Log_Pos: 511363808 Relay_Log_File: push-5-221-relay-bin.000034 Relay_Log_Pos: 382425277 Relay_Master_Log_File: master-223.000012 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 382425062 Relay_Log_Space: 511364283 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 2345 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2233306 Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592 Master_Info_File: /home/mysql-5.7.26/data/3307/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Reading event from the relay log Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec) ERROR: No query specified mysql>
Seconds_Behind_Master: 2345 发现从库同步延时很高,通过iotop查看
Total DISK READ : 0.00 B/s | Total DISK WRITE : 683.85 K/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 721.19 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 7061 be/4 mysql 0.00 B/s 153.28 K/s 0.00 % 72.27 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307 1766 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 8.51 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2730 be/4 mysql 0.00 B/s 11.79 K/s 0.00 % 7.50 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2164 be/4 mysql 0.00 B/s 11.79 K/s 0.00 % 7.47 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 8932 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 7.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 1840 be/4 mysql 0.00 B/s 35.37 K/s 0.00 % 7.14 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 936 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 6.77 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 1296 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 6.31 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 1201 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 5.32 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2728 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 5.05 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2539 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 5.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2531 be/4 mysql 0.00 B/s 47.16 K/s 0.00 % 4.21 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 6895 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 3.63 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307 2303 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 3.54 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 461 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 3.40 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2523 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 3.36 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2520 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 3.35 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2316 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 2.85 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2537 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 2.81 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2731 be/4 mysql 0.00 B/s 7.86 K/s 0.00 % 2.42 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 8940 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 2.07 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2310 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 1.91 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 8935 be/4 mysql 0.00 B/s 157.21 K/s 0.00 % 1.41 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 2302 be/4 mysql 0.00 B/s 3.93 K/s 0.00 % 1.30 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 6899 be/4 mysql 0.00 B/s 31.44 K/s 0.00 % 0.93 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307 32593 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.61 % [kworker/3:1] 7060 be/4 mysql 0.00 B/s 27.51 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql/3307/mysql.sock --port=3307 2535 be/4 mysql 0.00 B/s 15.72 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql/3306/mysql.sock --port=3306 13824 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd-udevd 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize 17
发现3307和3306实例占用IO最高,进入3307实例,查看sync_binlog变量
mysql> show variables like '%sync_binlog%' -> ; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | sync_binlog | 1 | +---------------+-------+ 1 row in set (0.01 sec) mysql>
发现设置的是1,也就是说每次事务提交都会将binlog的缓存写入磁盘,严重影响磁盘效率,将变量设置为1000,
mysql> set global sync_binlog=1000; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%sync_binlog%'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | sync_binlog | 1000 | +---------------+-------+ 1 row in set (0.00 sec)
再查看innodb_flush_log_at_trx_commit参数
mysql> show variables like '%innodb_flush_log%'; +--------------------------------+-------+ | Variable_name | Value | +--------------------------------+-------+ | innodb_flush_log_at_timeout | 1 | | innodb_flush_log_at_trx_commit | 1 | +--------------------------------+-------+ 2 rows in set (0.00 sec)
设置为1,也就是说每次事务提交,都会将innodb日志缓存写入磁盘,对磁盘效率影响很大,将它设置为2,每次事务提交时mysql都会把log buffer的数据写入log file,但是flush(刷到磁盘)操作并不会同时进行。该模式下,MySQL会每秒执行一次 flush(刷到磁盘)操作
mysql> set global innodb_flush_log_at_trx_commit=2; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%'; +--------------------------------+-------+ | Variable_name | Value | +--------------------------------+-------+ | innodb_flush_log_at_timeout | 1 | | innodb_flush_log_at_trx_commit | 2 | +--------------------------------+-------+ 2 rows in set (0.00 sec)
此时再查看从库同步状态
mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 172.28.5.223 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-223.000012 Read_Master_Log_Pos: 539225447 Relay_Log_File: push-5-221-relay-bin.000034 Relay_Log_Pos: 490154357 Relay_Master_Log_File: master-223.000012 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 490154142 Relay_Log_Space: 539225922 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 1378 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2233306 Master_UUID: 165a203c-8021-11e9-bc40-6c2b5992e592 Master_Info_File: /home/mysql-5.7.26/data/3307/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: System lock Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec) ERROR: No query specified
发现,同步延迟已经在很快缩小了,很快变为0,完全同步,这时通过iostat 查看io状态
[root@push-5-221 ~]# iostat -k -d -x 1 10/s 0.00 % 0.00 % [kworker/2:0H] Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.08 0.00 14.91 0.09 178.29 23.92 0.34 23.05 5.42 23.06 11.01 16.43 dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.40 12.69 0.24 dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00 dm-2 0.00 0.00 0.00 14.81 0.00 174.66 23.59 0.33 22.28 8.84 22.28 10.96 16.23 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 114.00 0.00 2013.00 35.32 3.14 27.40 0.00 27.40 8.75 99.80 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 113.00 0.00 2007.50 35.53 3.14 27.65 0.00 27.65 8.83 99.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 108.91 0.00 3026.24 55.57 3.10 27.85 0.00 27.85 9.05 98.51 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 111.88 0.00 3121.29 55.80 3.10 27.11 0.00 27.11 8.81 98.51 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 98.00 0.00 2648.00 54.04 2.81 29.63 0.00 29.63 9.71 95.20 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 95.00 0.00 2551.50 53.72 2.81 30.59 0.00 30.59 10.02 95.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 103.00 0.00 2037.00 39.55 2.97 28.67 0.00 28.67 9.12 93.90 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 104.00 0.00 2051.00 39.44 2.97 28.39 0.00 28.39 9.03 93.90
发现%util依旧差不多100%,磁盘IO性能还是很低,我们再把3306实例的变量设置改一下
[root@push-5-221 ~]# mysql -uroot -p -S /var/lib/mysql/3306/mysql.sock Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1476252 Server version: 5.7.26-log MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show variables like '%sync_binlog%'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | sync_binlog | 1 | +---------------+-------+ 1 row in set (0.01 sec) mysql> set global sync_binlog=1000; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%'; +--------------------------------+-------+ | Variable_name | Value | +--------------------------------+-------+ | innodb_flush_log_at_timeout | 1 | | innodb_flush_log_at_trx_commit | 1 | +--------------------------------+-------+ 2 rows in set (0.00 sec) mysql> set global innodb_flush_log_at_trx_commit=2; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%'; +--------------------------------+-------+ | Variable_name | Value | +--------------------------------+-------+ | innodb_flush_log_at_timeout | 1 | | innodb_flush_log_at_trx_commit | 2 | +--------------------------------+-------+ 2 rows in set (0.00 sec) mysql>
此时,再通过iostat查看IO
[root@push-5-221 ~]# iostat -k -d -x 1 10 Linux 3.10.0-957.el7.x86_64 (push-5-221) 2019年07月05日 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.08 0.00 14.92 0.09 178.36 23.92 0.34 23.06 5.42 23.06 11.01 16.43 dm-0 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.39 12.69 0.24 dm-1 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00 dm-2 0.00 0.00 0.00 14.81 0.00 174.72 23.59 0.33 22.28 8.84 22.28 10.96 16.23 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 21.00 0.00 550.00 52.38 0.34 16.19 0.00 16.19 13.76 28.90 dm-0 0.00 0.00 0.00 2.00 0.00 1.50 1.50 0.05 24.00 0.00 24.00 24.00 4.80 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 19.00 0.00 548.50 57.74 0.29 15.37 0.00 15.37 12.74 24.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00
%util已经顺利降到40%左右了。
转:https://www.cnblogs.com/sky-cheng/p/11136943.html