17、【SparkSQL】org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism异常

  • 背景
    当我在使用IDEA本地连接服务器spark服务时,虽然拷贝了hive-site.xml,提供了相应的依赖,仍然报错:org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism
    依赖如下:

      <dependencies>
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-core_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-sql_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>mysql</groupId>
              <artifactId>mysql-connector-java</artifactId>
              <version>8.0.25</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.hive</groupId>
              <artifactId>hive-exec</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-hive_2.12</artifactId>
              <version>3.1.2</version>
          </dependency>
    
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-auth</artifactId>
              <version>3.2.0</version>
          </dependency>
      </dependencies>
    
    
  • 分析
    在远程访问hadoop时,需要提供认证信息

    https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html
    https://blog.csdn.net/csr_hema/article/details/8147590

  • 措施
    在Maven添加如下依赖:

            <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-auth</artifactId>
              <version>3.2.0</version>
          </dependency>
    
上一篇:word文档的图片怎么保存到百度Web编辑器上


下一篇:ORACLE-常用基础命令总结