Apache Doris ODBC Mysql外表在Ubuntu下使用方法及配置

2021-12-18 04:22:16

今天是2021年最后一天，提前祝大家元旦快乐，这也是2021年最后一篇文章，后续会给大家带来更多关于Doris的文章，同时也希望Doris 2022年起飞，顺利从Apache 孵化器毕业成*项目，给大家带来更快、更稳定、生态更丰富的MPP OLAP分析型数据库产品。

Apache Doris 社区 2022 年的总体规划，包括待开展或已开展、以及已完成但需要持续优化的功能、文档、社区建设等多方面，我们期待有更多的小伙伴参与进来讨论。同时也希望多多关注Doris，给Doris加Star

Doris 2022 Roadmap

之前写了在Centos下Apache Doris 外表使用方法及注意实现，但是不少用户是使用ubuntu系统，在这个系统下很多用户遇到ODBC外表导致BE服务宕机的问题，对此我专门在ubuntu 18.04下进行了测试

1.软件环境

  1. 操作系统：ubuntu 18.04
  2. Apache Doris ：0.15
  3. mysql 数据库：8.0.27-0ubuntu0.20.04.1 for Linux on x86_64
  4. UnixODBC：2.3.4
  5. Mysql Connector ODBC ：5.3.13、8.0.11、8.0.26

2.安装ODBC驱动

首先我们安装unixODBC驱动、这里直接给出驱动的下载地址及安装命令

sudo wget ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.4.tar.gz 
tar -xvzf unixODBC-2.3.4.tar.gz 
cd unixODBC-2.3.4/
sudo ./configure --prefix=/usr/local/unixODBC-2.3.7 --includedir=/usr/include --libdir=/usr/lib -bindir=/usr/bin --sysconfdir=/etc
make
sudo make install
​
​

安装成功后，unixODBC所需的头文件都被安装到了/usr/inlucde下，编译好的库文件安装到了/usr/lib下，与unixODBC相关的可执行文件安装到了/usr/bin下，配置文件放到了/etc下。

验证安装是否成功

# odbcinst -j
unixODBC 2.3.4
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8

3.安装Mysql ODBC驱动

这里我默认你是知道Mysql的安装方法，或者你已经有了Mysql数据库，对Mysql的安装配置就不在讲了，如果这块不清楚，请去百度。

从mysql 站点下载对应的驱动

https://dev.mysql.com/downloads/connector/odbc/

我下载的是8.0.26、8.0.11、5.3.13

mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit.tar.gz

下载后解压，我这里为了测试方便，直接将解压后的目录重命名使用了，没有将这个目录下的bin和lib拷贝到/usr/local/目录下，具体的操作命令：

tar zxvf mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit.tar.gz
mv mysql-connector-odbc-8.0.26-linux-glibc2.12-x86-64bit mysql-odbc-8.0.26

注册Mysql驱动

myodbc-installer -a -d -n "MySQL ODBC 8.0.26 Unicode Driver" -t "Driver=/root/mysql-odbc-8.0.26/lib/libmyodbc8w.so"
myodbc-installer -a -d -n "MySQL ODBC 8.0.26 ANSI Driver" -t "Driver=/root/mysql-odbc-8.0.26/lib/libmyodbc8a.so"

然后查看是否注册成功

使用这个命令：myodbc-installer -d -l

MySQL ODBC 5.0 Unicode Driver
MySQL ODBC 5.0 ANSI Driver
MySQL ODBC 8.0 Unicode Driver
MySQL ODBC 8.0
MySQL ODBC 8.0.26 Unicode Driver
MySQL ODBC 8.0.26 ANSI Driver

这里我安装了上面说的三个版本的驱动，另外两个版本的驱动安装方式一样

4.验证通过ODBC访问Mysql

我们去配置ODBC访问Mysql的参数

编辑/etc/odbc.ini文件，加入下面的内容，将信息替换成你自己的

[mysql]
Description     = Data source MySQL
Driver          = MySQL ODBC 8.0 Unicode Driver
Server          = localhost
Host            = localhost
Database        = demo
Port            = 3306
User            = root
Password        = zhangfeng

然后我们通过：

isql -v mysql

isql -v mysql
+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
​

说明我们ODBC配置成功

5.测试Apache Doris ODBC外表

Doris的安装配置参考我的博客：Apache Doris安装部署，或者官网的文档

首先我们在Mysql数据库见了一个demo库及相应的表

CREATE TABLE `test_cdc` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=91234 DEFAULT CHARSET=utf8mb4;

对应的测试数据

INSERT INTO `test_cdc` VALUES (123, 'this is a update');
INSERT INTO `test_cdc` VALUES (1212, '测试flink CDC');
INSERT INTO `test_cdc` VALUES (1234, '这是测试');
INSERT INTO `test_cdc` VALUES (11233, 'zhangfeng_1');
INSERT INTO `test_cdc` VALUES (21233, 'zhangfeng_2');
INSERT INTO `test_cdc` VALUES (31233, 'zhangfeng_3');
INSERT INTO `test_cdc` VALUES (41233, 'zhangfeng_4');
INSERT INTO `test_cdc` VALUES (51233, 'zhangfeng_5');
INSERT INTO `test_cdc` VALUES (61233, 'zhangfeng_6');
INSERT INTO `test_cdc` VALUES (71233, 'zhangfeng_7');
INSERT INTO `test_cdc` VALUES (81233, 'zhangfeng_8');
INSERT INTO `test_cdc` VALUES (91233, 'zhangfeng_9');

下面建Doris的ODBC外表，这里我们是通过ODBC_Resource来创建ODBC外表，这也是推荐的方式，这样ODBC resource是可以复用

首先我们在BE节点的conf/odbcinst.ini，添加我们的刚才注册的8.0.26的ODBC驱动（[MySQL ODBC 8.0.26]这部分）

# Driver from the postgresql-odbc package
# Setup from the unixODBC package
[PostgreSQL]
Description     = ODBC for PostgreSQL
Driver          = /usr/lib/psqlodbc.so
Setup           = /usr/lib/libodbcpsqlS.so
FileUsage       = 1
​
​
# Driver from the mysql-connector-odbc package
# Setup from the unixODBC package
[MySQL ODBC 8.0 Unicode Driver]
Description     = ODBC for MySQL
Driver          = /usr/lib64/libmyodbc8w.so
FileUsage       = 1
​
[MySQL ODBC 5.0 Unicode Driver]
Description     = ODBC for MySQL
Driver=/root/mysql-odbc/lib/libmyodbc5w.so
FileUsage       = 1
​
[MySQL ODBC 8.0]
Description     = ODBC for MySQL
Driver=/root/mysql-odbc-8/lib/libmyodbc8w.so
FileUsage       = 1
​
[MySQL ODBC 8.0.26]
Description     = ODBC for MySQL
Driver=/root/mysql-odbc-8.0.26//lib/libmyodbc8w.so
FileUsage       = 1
​
​
# Driver from the oracle-connector-odbc package
# Setup from the unixODBC package
[Oracle 19 ODBC driver]
Description=Oracle ODBC driver for Oracle 19
Driver=/usr/lib/libsqora.so.19.1

首先我们创建resource

 CREATE EXTERNAL RESOURCE `mysql_8_0_26`
 PROPERTIES (
"host" = "localhost",
 "port" = "3306",
 "user" = "root",
 "password" = "zhangfeng",
 "database" = "demo",
 "table" = "test_cdc",
 "driver" = "MySQL ODBC 8.0.26",  --注意这里的名称要和我们上面红框标识部分的[]里的名称一致
 "odbc_type" = "mysql",
 "type" = "odbc_catalog"
 );

基于这个resource创建ODBC外表

CREATE EXTERNAL TABLE `test_odbc_8_0_26` (
  `id` int NOT NULL ,
  `name` varchar(255) null
) ENGINE=ODBC
COMMENT "ODBC"
PROPERTIES (
"odbc_catalog_resource" = "mysql_8_0_26", --这里的名称就是我们上面定义的resource的名称
"database" = "demo",
"table" = "test_cdc"
);

执行结果

mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
​
Database changed
mysql> drop table test_odbc_8_0_26;
Query OK, 0 rows affected (0.01 sec)
​
mysql>  CREATE EXTERNAL RESOURCE `mysql_8_0_26`
    ->  PROPERTIES (
    -> "host" = "localhost",
    ->  "port" = "3306",
    ->  "user" = "root",
    ->  "password" = "zhangfeng",
    ->  "database" = "demo",
    ->  "table" = "test_cdc",
    ->  "driver" = "MySQL ODBC 8.0.26",
    ->  "odbc_type" = "mysql",
    ->  "type" = "odbc_catalog"
    ->  );
Query OK, 0 rows affected (0.01 sec)
mysql>  CREATE EXTERNAL TABLE `test_odbc_8_0_26` (
    ->   `id` int NOT NULL ,
    ->   `name` varchar(255) null
    -> ) ENGINE=ODBC
    -> COMMENT "ODBC"
    -> PROPERTIES (
    -> "odbc_catalog_resource" = "mysql_8_0_26",
    -> "database" = "demo",
    -> "table" = "test_cdc"
    -> );
Query OK, 0 rows affected (0.01 sec)

一切显示正常，下面是见证奇迹的时候，我们执行查询外表操作

select * from test_odbc_8_0_26;
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason)]

这个时候显示BE节点挂了，我们通过show backends命令去查看也是显示BE节点挂了

mysql> show backends\G;
*************************** 1. row ***************************
            BackendId: 10002
              Cluster: default_cluster
                   IP: 172.16.192.81
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2021-12-31 10:43:16
        LastHeartbeat: 2021-12-31 11:03:00
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 0
     DataUsedCapacity: 0.000
        AvailCapacity: 83.224 GB
        TotalCapacity: 98.305 GB
              UsedPct: 15.34 %
       MaxDiskUsedPct: 15.34 %
                  Tag: {"location" : "default"}
               ErrMsg:
              Version: 0.15.1-rc09-Unknown
               Status: {"lastSuccessReportTabletsTime":"2021-12-31 11:02:22","lastStreamLoadTime":-1}
1 row in set (0.00 sec)

查看BE的日志

显示是因为ODBC问题导致BE宕机，因为之前我在centos是没问题的（但是版本不是这个是8.0.11），这里考虑可能是版本不匹配的问题。后来询问了社区这块的开发者，得到的答案也确实是这个问题，那么我们换一个版本试试，这个时候我首先想到的我不在使用8.x版本的驱动，使用5.x试试，关于5.x的驱动安装，我就不在详细讲解了，和8.x的安装方法一样，参考上面的部分

*** Aborted at 1640918068 (unix time) try "date -d @1640918068" if you are using GNU date ***
PC: @     0x7f8caaf29b7e (unknown)
*** SIGSEGV (@0x0) received by PID 56420 (TID 0x7f8c62370700) from PID 0; stack trace: ***
    @          0x3022682 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f8cab0f93c0 (unknown)
    @     0x7f8caaf29b7e (unknown)
    @          0x34ef4ac getrn
    @          0x34ef722 lh_insert
    @          0x34abd14 OBJ_NAME_add
    @     0x7f8c09f9d115 ossl_init_ssl_base_ossl_
    @     0x7f8cab0f647f __pthread_once_slow
    @     0x7f8c0a3af194 CRYPTO_THREAD_run_once
    @     0x7f8c09f9cf87 OPENSSL_init_ssl
    @     0x7f8c0a77e504 ssl_start()
    @     0x7f8c0a751f55 mysql_server_init
    @     0x7f8c0a75c425 mysql_init
    @     0x7f8c0a72cbcd DBC::connect()
    @     0x7f8c0a72f403 MySQLDriverConnect()
    @     0x7f8c0a74ea08 SQLDriverConnectW
    @          0x3b060ec SQLDriverConnect
    @          0x1ec046b doris::ODBCConnector::open()
    @          0x1eb9706 doris::OdbcScanNode::open()
    @          0x189e6e9 doris::PlanFragmentExecutor::open_internal()
    @          0x189fb4c doris::PlanFragmentExecutor::open()
    @          0x181b70e doris::FragmentExecState::execute()
    @          0x181f706 doris::FragmentMgr::_exec_actual()
    @          0x1828d4f std::_Function_handler<>::_M_invoke()
    @          0x198a963 doris::ThreadPool::dispatch_thread()
    @          0x1984aac doris::Thread::supervise_thread()
    @     0x7f8cab0ed609 start_thread
    @     0x7f8caaec5293 clone
    @                0x0 (unknown)

下面我们开始测试5.3.13版本的驱动对Apache Doris外表是否正常，我们首先启动这个BE节点

关于5.3.13的ODBC驱动安装、注册及Doris BE节点的配置请参考上面的部分，我们直接进入测试环节

从下图的测试来看一切正常

mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
​
Database changed
mysql>  CREATE EXTERNAL RESOURCE `mysql_5`
    ->  PROPERTIES (
    -> "host" = "localhost",
    ->  "port" = "3306",
    ->  "user" = "root",
    ->  "password" = "zhangfeng",
    ->  "database" = "demo",
    ->  "table" = "test_cdc",
    ->  "driver" = "MySQL ODBC 5.0 Unicode Driver",
    ->  "odbc_type" = "mysql",
    ->  "type" = "odbc_catalog"
    ->  );
Query OK, 0 rows affected (0.01 sec)
​
mysql>
mysql>
mysql>  CREATE EXTERNAL TABLE `test_odbc_5` (
    ->   `id` int NOT NULL ,
    ->   `name` varchar(255) null
    -> ) ENGINE=ODBC
    -> COMMENT "ODBC"
    -> PROPERTIES (
    -> "odbc_catalog_resource" = "mysql_5",
    -> "database" = "demo",
    -> "table" = "test_cdc"
    -> );
Query OK, 0 rows affected (0.01 sec)
​
mysql> select * from `test_odbc_5`;
+-------+------------------+
| id    | name             |
+-------+------------------+
|   123 | this is a update |
|  1212 | 测试flink CDC    |
|  1234 | 这是测试         |
| 11233 | zhangfeng_1      |
| 21233 | zhangfeng_2      |
| 31233 | zhangfeng_3      |
| 41233 | zhangfeng_4      |
| 51233 | zhangfeng_5      |
| 61233 | zhangfeng_6      |
| 71233 | zhangfeng_7      |
| 81233 | zhangfeng_8      |
| 91233 | zhangfeng_9      |
+-------+------------------+
12 rows in set (0.01 sec)

这个时候我在想，我在centos下8.0.11都是正常的，在ubuntu下这个版本是不是也是正常的呢，所以我又去测试了8.0.11这个版本，从测试来看也是正常的，其他的我就没再去做测试，可能是8.0.26这个版本太新了，还有就是Linux的各种都需要手动配置，这就需要你对这些非常熟悉，针对Doris这个ODBC外表问题出现异常，基本都是你使用的驱动版本不对，换个版本在重试

最后针对Ubuntu下的Apache Doris Mysql ODBC外表的使用Mysql ODBC驱动的版本，推荐：5.3.13和8.0.11这两个版本，你也可以自己尝试其他版本

上一篇：perl DBI动态切换连接数据源


下一篇：模拟iic思路参考