1、下载(建议使用github上的)
http://maxwells-daemon.io/quickstart/ # version > 1.30 jdk需要使用11版本 https://github.com/zendesk/maxwell/releases # version <1.30 jdk使用8版本
2、解压
tar -xf xxx.tar.gz
3、运行示例 http://maxwells-daemon.io/quickstart/ 。查看帮助信息:bin/maxwell --help
bin/maxwell --user=‘maxwell‘ --password=‘XXXXXX‘ --host=‘127.0.0.1‘ --producer=kafka --kafka.bootstrap.servers=localhost:9092 --kafka_topic=maxwell
4、通过config.properties运行 (增量监听)
#mysql user=root password=yang156122 host=hadoop101 #kafka producer=kafka kafka.bootstrap.servers=hadoop101:9092 kafka_topic=kafkaeagle1
../bin/maxwell --config config.properties
{"database":"test","table":"user","type":"update","ts":1627615996,"xid":302686,"commit":true,"data":{"id":12,"name":"324223","age":2.00,"create_time":"2021-07-23 13:36:53"},"old":{"name":"324223490"}} #update {"database":"test","table":"user","type":"delete","ts":1627616012,"xid":302732,"commit":true,"data":{"id":12,"name":"324223","age":2.00,"create_time":"2021-07-23 13:36:53"}} # delete {"database":"test","table":"user","type":"insert","ts":1627616033,"xid":302789,"commit":true,"data":{"id":12,"name":"2324324","age":1232.00,"create_time":"2021-07-30 11:33:28"}} # insert
5、全量监听(使用的就是select取数); 注意:要确保步骤四步在运行 。查看帮助信息:./bin/maxwell-bootstrap --help
./bin/maxwell-bootstrap --user=‘root‘ --password=‘yang156122‘ --host=‘hadoop101‘ --database=test --table=user_copy2 --client_id maxwell ##全量
kafka数据格式:
{"database":"maxwell","table":"bootstrap","type":"insert","ts":1627615948,"xid":302392,"commit":true,"data":{"id":3,"database_name":"test","table_name":"user_copy2","where_clause":null,"is_complete":0,"inserted_rows":0,"total_rows":6,"created_at":null,"started_at":null,"completed_at":null,"binlog_file":null,"binlog_position":0,"client_id":"maxwell","comment":null}} {"database":"test","table":"user_copy2","type":"bootstrap-start","ts":1627615949,"data":{}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":1,"name":"name1","age":1.00,"create_time":"2021-07-30 23:48:17"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":2,"name":"tablename","age":1.00,"create_time":"2021-07-02 05:33:26"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":3,"name":"name33","age":333.00,"create_time":"2021-07-02 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":4,"name":"4name4","age":4.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":5,"name":"5name6","age":5.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627615949,"data":{"id":6,"name":"6name632","age":6.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-complete","ts":1627615949,"data":{}}
6、全量监听(使用的就是select取数,带where条件取数); 注意:要确保步骤四步在运行
./bin/maxwell-bootstrap --user=‘root‘ --password=‘yang156122‘ --host=‘hadoop101‘ --database=test --table=user_copy2 --client_id maxwell --where ‘id>=3‘
kafka数据格式:
{"database":"maxwell","table":"bootstrap","type":"insert","ts":1627616689,"xid":306661,"commit":true,"data":{"id":13,"database_name":"test","table_name":"user_copy2","where_clause":"id>=3","is_complete":0,"inserted_rows":0,"total_rows":4,"created_at":null,"started_at":null,"completed_at":null,"binlog_file":null,"binlog_position":0,"client_id":"maxwell","comment":null}} {"database":"test","table":"user_copy2","type":"bootstrap-start","ts":1627616690,"data":{}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627616690,"data":{"id":3,"name":"name33","age":333.00,"create_time":"2021-07-02 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627616690,"data":{"id":4,"name":"4name4","age":4.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627616690,"data":{"id":5,"name":"5name6","age":5.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627616690,"data":{"id":6,"name":"6name632","age":6.00,"create_time":"2021-07-05 06:44:58"}} {"database":"test","table":"user_copy2","type":"bootstrap-complete","ts":1627616690,"data":{}}
./bin/maxwell-bootstrap --user=‘root‘ --password=‘yang156122‘ --host=‘hadoop101‘ --database=test --table=user_copy2 --client_id maxwell --where "create_time >= ‘2021-07-05‘"
{"database":"maxwell","table":"bootstrap","type":"insert","ts":1627616966,"xid":308581,"commit":true,"data":{"id":19,"database_name":"test","table_name":"user_copy2","where_clause":"create_time >= ‘2021-07-05‘","is_complete":0,"inserted_rows":0,"total_rows":1,"created_at":null,"started_at":null,"completed_at":null,"binlog_file":null,"binlog_position":0,"client_id":"maxwell","comment":null}} {"database":"test","table":"user_copy2","type":"bootstrap-start","ts":1627616966,"data":{}} {"database":"test","table":"user_copy2","type":"bootstrap-insert","ts":1627616966,"data":{"id":1,"name":"name1","age":1.00,"create_time":"2021-07-30 23:48:17"}} {"database":"test","table":"user_copy2","type":"bootstrap-complete","ts":1627616966,"data":{}}
FAQ
错误一:
(base) [root@hadoop101 maxwell-1.33.1]# bin/maxwell --user=‘root‘ --password=‘yang156122‘ --host=‘hadoop101‘ > --producer=kafka --kafka.bootstrap.servers=hadoop101:9092 --kafka_topic=kafkaeagle1 Using kafka version: 1.0.0 Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.UnsupportedClassVersionError: com/zendesk/maxwell/Maxwell has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:601)
解决方法:maxwell1.33版本要使用jdk11,因此要么就升级jdk,要么就使用maxwell低版本(小于1.30版本)