tachyon with spark

2022-10-29 23:18:22

spark1.2.0 tachyon0.5.0 jdk1.7 scala2.10.4

1.装好spark、tachyon、jdk、scala

2.修改spark-env.sh添加Tachyon客户端jar包环境变量

export HADOOP_CLASSPATH=/home/hadoop/tachyon-0.5.-bin/client/target/tachyon-cli

ent-0.5.-jar-with-dependencies.jar

3.创建一个core-site.xml并配置（本没有，自己创建）

<configuration>

  <property>

      <name>fs.tachyon.impl</name>

     <value>tachyon.hadoop.TFS</value>

  </property>

</configuration>

4.启动spark启动tachyon

tachyon-start.sh local

sbin/start-all.sh

5.测试

spark-shell

val s =sc.textFile("tachyon://hadoop:19998/test/passwd")

s.count

s.saveAsTextFile("tachyon://hadoop:19998/count")

6.查看结果

码农公寓