scala - Importing a text file into Cassandra using Spark when there are multiple variable types -
i'm using spark import info text files cql tables (on datastax). i've done 1 file in variables strings. first created table using cql, in spark shell using scala ran:
val file = sc.textfile("file:///home/pr.txt").map(line => line.split("\\|").map(_.tostring)); file.map(line => (line(0), line(1))).savetocassandra("ks", "ks_pr", seq("proc_c", "proc_d"));
the rest of files want import contain multiple variable types. i've set tables using cql , specified appropriate types there, how transform them when importing text file in spark?
for illustration if proc_c int , proc_d double can way:
file.map{ line => (line(0), line(1)). map({ case (l, r) => (l.toint, r.todouble) }). savetocassandra("ks", "ks_pr", seq("proc_c", "proc_d") }
scala cassandra apache-spark cql datastax-enterprise
No comments:
Post a Comment