IDEA本地执行Spark报错:is not a valid DFS filename
本地执行spark structured streaming 报错,程序代码:def main(args: Array[String]): Unit = {val spark = SparkSession.builder.master("local[2]").appName("sparkStream2hudi").getOrCreate()//消费kafkaimport spark.implici
·
本地执行spark structured streaming 报错,程序代码:
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder
.master("local[2]")
.appName("sparkStream2hudi")
.getOrCreate()
//消费kafka
import spark.implicits._
val df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "192.168.78.12:9092")
.option("subscribe", "test1")
.load()
//打印输出
val out = df.selectExpr("CAST(value AS STRING)")
.as[String]
.writeStream
.outputMode("append")
.format("console")
.start()
out.awaitTermination()
}
项目资源文件:
错误日志:
22/02/21 10:24:07 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
22/02/21 10:24:10 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
Exception in thread "main" java.lang.IllegalArgumentException: Pathname /C:/Users/lixz/AppData/Local/Temp/temporary-fde61567-9663-47b7-ba34-492e1d3d2872 from C:/Users/lixz/AppData/Local/Temp/temporary-fde61567-9663-47b7-ba34-492e1d3d2872 is not a valid DFS filename.
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:243)
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1321)
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1318)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1335)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1310)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326)
at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:90)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.<init>(MicroBatchExecution.scala:48)
at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:281)
at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:322)
at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:325)
at kl.stream.流式写hudi测试$.main(流式写hudi测试.scala:29)
at kl.stream.流式写hudi测试.main(流式写hudi测试.scala)
22/02/21 10:24:10 INFO spark.SparkContext: Invoking stop() from shutdown hook
解决方案:
由于项目资源文件中包含了我们生产环境hadoop集群的配置文件core-site.xml和hdfs-site.xml,程序本地执行时会读取这些配置文件,文件中的路径是unix路径,windows并不支持这种格式,所以会报出错误,所以本地调试时可以移除掉这两个配置文件
更多推荐
已为社区贡献4条内容
所有评论(0)