Wednesday 15 August 2012

scala - Importing a text file into Cassandra using Spark when there are multiple variable types -



scala - Importing a text file into Cassandra using Spark when there are multiple variable types -

i'm using spark import info text files cql tables (on datastax). i've done 1 file in variables strings. first created table using cql, in spark shell using scala ran:

val file = sc.textfile("file:///home/pr.txt").map(line => line.split("\\|").map(_.tostring)); file.map(line => (line(0), line(1))).savetocassandra("ks", "ks_pr", seq("proc_c", "proc_d"));

the rest of files want import contain multiple variable types. i've set tables using cql , specified appropriate types there, how transform them when importing text file in spark?

for illustration if proc_c int , proc_d double can way:

file.map{ line => (line(0), line(1)). map({ case (l, r) => (l.toint, r.todouble) }). savetocassandra("ks", "ks_pr", seq("proc_c", "proc_d") }

scala cassandra apache-spark cql datastax-enterprise

No comments:

Post a Comment