数据迁移上云优化之 - 带宽压缩加速

现在云已经是一个趋势毋庸置疑,但是怎样在有限的时间内把数据弄上云端是个需要考虑的问题。
如果能增量复制,整个时间窗口可以不需要太关注,因为切换的时间窗口比较短。
但是如果是全量迁移,或者增量迁移前的全量迁移,则需要考虑时间窗口。
一般来说,从线下到线上,网络带宽都比较低,这个我在以前把香港机房的数据迁移到国内机房时深有同感,速度之慢难以忍受。
下面是一套加速方案:
数据迁移上云优化之 - 带宽压缩加速
数据迁移上云优化之 - 带宽压缩加速
数据迁移上云优化之 - 带宽压缩加速
数据迁移上云优化之 - 带宽压缩加速
数据迁移上云优化之 - 带宽压缩加速
数据迁移上云优化之 - 带宽压缩加速
下面是一个例子:
首先在本地服务器建立远程数据库服务器隧道 :

ssh -C -L 6666:127.0.0.1:1921 postgres@remote_ip  

在另外一个本地的Shell里面执行如下命令,备份远程数据库的数据到本地:

postgres@db-172-16-3-33-> date +%F%T;psql -h 127.0.0.1 -p 6666 -U postgres db_remote -c "copy schema.tbl_log to stdout" >./tbl_log.dmp;date +%F%T  
2011-06-2717:16:50  
2011-06-2717:17:08  

耗时18秒

不走隧道的话

postgres@db-172-16-3-33-> date +%F%T;psql -h remote_ip -p 1921 -U schema db_remote -c "copy schema.tbl_log to stdout" >./tbl_log2.dmp;date +%F%T  
2011-06-2717:20:06  
2011-06-2717:22:29  

耗时143秒

速度提升非常惊人.

参考:

man ssh
     -L [bind_address:]port:host:hostport  
             Specifies that the given port on the local (client) host is to be forwarded to the given host and port on  
             the remote side.  This works by allocating a socket to listen to port on the local side, optionally bound  
             to the specified bind_address.  Whenever a connection is made to this port, the connection is forwarded  
             over the secure channel, and a connection is made to host port hostport from the remote machine.  Port  
             forwardings can also be specified in the configuration file.  IPv6 addresses can be specified with an  
             alternative syntax: [bind_address/]port/host/hostport or by enclosing the address in square brackets.  
             Only the superuser can forward privileged ports.  By default, the local port is bound in accordance with  
             the GatewayPorts setting.  However, an explicit bind_address may be used to bind the connection to a spe-  
             cific address.  The bind_address of “localhost” indicates that the listening port be bound for local use  
             only, while an empty address or ‘*’ indicates that the port should be available from all interfaces.  
  
     -C      Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP  
             connections).  The compression algorithm is the same used by gzip(1), and the “level” can be controlled  
             by the CompressionLevel option for protocol version 1.  Compression is desirable on modem lines and other  
             slow connections, but will only slow down things on fast networks.  The default value can be set on a  
             host-by-host basis in the configuration files; see the Compression option.  
上一篇:说说delete与truncate的区别?


下一篇:Android自定义组合控件---教你如何自定义下拉刷新和左滑删除