memsql filesystem pipeline 试用

一些功能类似drill ,比如s3,file 。。。

创建file pipeline

  • 准备file
mkdir -p /opt/db/
touch books.txt
内容如下:
The Catcher in the Rye, J.D. Salinger, 1945
Pride and Prejudice, Jane Austen, 1813
Of Mice and Men, John Steinbeck, 1937
Frankenstein, Mary Shelley, 1818
  • 创建表
memsql

CREATE DATABASE books;

USE books;

CREATE TABLE classic_books
(
title VARCHAR(255),
author VARCHAR(255),
date VARCHAR(255)
);
  • 创建pipeline
CREATE PIPELINE library
AS LOAD DATA FS '/opt/db/*'
INTO TABLE `classic_books`
FIELDS TERMINATED BY ',';

启用pipeline

  • 启动
START PIPELINE library;
  • 查看状态
SHOW PIPELINES;

测试结果

memsql filesystem pipeline 试用
memsql filesystem pipeline 试用

几个问题

  • Paused due to error. Run START PIPELINE or consider setting pipelines_stop_on_error to false
配置参数修改
SET GLOBAL pipelines_stop_on_error = false;
  • 注意文件的权限,同时文件必须是每个节点的,不然一直看不到数据(我没注意一直在master,就有问题)
  • 经常有以下提示
Data volume has significantly changed since the last time ANALYZE TABLE was run. Run <a target="_blank" href="https://docs.memsql.com/ops-redir/analyze/?utm_source=ops&amp;utm_medium=link&amp;utm_campaign=ref" data-reactid=".0.3.1.2.0.2.0.0.0.3" style="background:rgb(250, 250, 250);color:rgb(18, 135, 186);text-decoration:none;font-size:1.067rem;"><b data-reactid=".0.3.1.2.0.2.0.0.0.3.0">ANALYZE TABLE</b></a> on each table to improve query performance andrefresh schema.

解决方法

按照提示操作即可,这个可能和我的系统没有进行参数优化有关,具体的可以参考下面的资料的安装最佳实践

参考资料

https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-quickstart/
https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-overview/
https://docs.memsql.com/tutorials/v6.0/installation-best-practices/

 
 
 
 
上一篇:python学习笔记_week9


下一篇:聊聊我怎么系统学习Linux技能并快速提高的