ICode9

精准搜索请尝试: 精确搜索
首页 > 数据库> 文章详细

Spark SQL 数据源 parquet文件

2022-06-24 17:04:57  阅读:44  来源: 互联网

标签:string scala 数据源 parquet sql apache org Spark


scala> val employee = sqlparquet.read.json("employee.json") 这里将txt转化为parquet应该也行
employee: org.apache.spark.sql.DataFrame = [_corrupt_record: string, age: string ... 2 more fields]

scala> employee.write.parquet("employee.parquet")
                                                                                
scala> val sqlpar = new org.apache.spark.sql.SQLContext(sc)
warning: one deprecation (since 2.0.0); for details, enable `:setting -deprecation' or `:replay -deprecation'
sqlpar: org.apache.spark.sql.SQLContext = [email protected]

scala> val parread = sqlpar.read.parquet("employee.parquet")
parread: org.apache.spark.sql.DataFrame = [_corrupt_record: string, age: string ... 2 more fields]

scala> parread.show()
此处虽然可以输出但是没在表中,这里属于parquet文件读取
scala> val allcol = sqlpar.sql("SELECT * FROM Demo")
allcol: org.apache.spark.sql.DataFrame = [_corrupt_record: string, age: string ... 2 more fields]
scala> val allcol = sqlpar.sql("SELECT id,age,name FROM Demo")
allcol: org.apache.spark.sql.DataFrame = [id: string, age: string ... 1 more field]

scala> allcol.show()
+----+----+-------+
|  id| age|   name|
+----+----+-------+
|null|null|   null|
|1201|  25| satish|
|1202|  28|krishna|
|1203|  39|  amith|
|1204|  23|  javed|
|1205|  23| prudvi|
|null|null|   null|
+----+----+-------+
此处为存在临时表中用sql读表

后续补充json. hive. paruqet三种数据源优缺点

标签:string,scala,数据源,parquet,sql,apache,org,Spark
来源: https://www.cnblogs.com/19951009Z/p/16409159.html

专注分享技术,共同学习,共同进步。侵权联系[[email protected]]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有