Can I treat case class as a Dataframe?

In other languages such as python (pyspark) and R, we have the dataframe types.
In scala we have the case class. May I think their behavior are similar?

Such as this:

case class Employee (
  name: String,
  title: String,
  salary: Double,
  rate: Double,
  insurance: Double)

val employees = List (
  Employee("buck trends","ceo",200000,0.25,100),
  Employee("cindy banks","cfo",170000,0.22,120),
  Employee("joe coder","developer",130000,0.20,120))

to spark:

>>> df = spark.createDataFrame(list,["name","title","salary","rate","insurance"])
|       name|    title|salary|rate|insurance|
|buck trends|      ceo|200000|0.25|      100|
|cindy banks|      cfo|170000|0.22|      120|
|  joe coder|developer|130000| 0.2|      120|

Thank you.

I don’t understand the question?

A case class can’t be a dataframe, since it only represents a single record.
A List of case classes can or not be considered a dataframe, depending on what you expect of it.

I personally have a lot of success processing data using fs2.Stream[IO, SomeCaseClass] but I also know a lot of people that say that Scala is missing a proper n-dimensional array and dataframe libraries like numpy and pandas for when yu don’t want to pull out an extremely big dependency like Spark.

1 Like

hello @BalmungSan

In scala with what syntax i can know the object’s attribute?
such as in the sample above, how can I know ‘employees’ is a case class object?
And ‘a’ is a char, “a” is a string, etc.

Such as this in ruby:

irb(main):001:0> "a".class
=> String

And, if you know please kindly tell me what’s the method to check a object’s all members. such as this:

irb(main):009:0> "a".methods.sort
=> [:!, :!=, :!~, :%, :*, :+, :+@, :-@, :<, :<<, :<=, :<=>, :==, :===, :=~, :>, :>=, :[], :[]=, :__id__, :__send__, :ascii_only?, :b, :between?, :bytes, :bytesize, :byteslice, :capitalize, :capitalize!, :casecmp, :casecmp?, :center, :chars, :chomp, :chomp!, :chop, :chop!, :chr, :clamp, :class, :clear, :clone, :codepoints, :concat, :count, :crypt, :define_singleton_method, :delete, :delete!, :delete_prefix, :delete_prefix!, :delete_suffix, :delete_suffix!, :display, :downcase, :downcase!, :dump, :dup, :each_byte, :each_char, :each_codepoint, :each_grapheme_cluster, :each_line, :empty?, :encode, :encode!, :encoding, :end_with?, :enum_for, :eql?, :equal?, :extend, :force_encoding, :freeze, :frozen?, :getbyte, :grapheme_clusters, :gsub, :gsub!, :hash, :hex, :include?, :index, :insert, :inspect, :instance_eval, :instance_exec, :instance_of?, :instance_variable_defined?, :instance_variable_get, :instance_variable_set, :instance_variables, :intern, :is_a?, :itself, :kind_of?, :length, :lines, :ljust, :lstrip, :lstrip!, :match, :match?, :method, :methods, :next, :next!, :nil?, :object_id, :oct, :ord, :partition, :prepend, :private_methods, :protected_methods, :public_method, :public_methods, :public_send, :remove_instance_variable, :replace, :respond_to?, :reverse, :reverse!, :rindex, :rjust, :rpartition, :rstrip, :rstrip!, :scan, :scrub, :scrub!, :send, :setbyte, :singleton_class, :singleton_method, :singleton_methods, :size, :slice, :slice!, :split, :squeeze, :squeeze!, :start_with?, :strip, :strip!, :sub, :sub!, :succ, :succ!, :sum, :swapcase, :swapcase!, :taint, :tainted?, :tap, :then, :to_c, :to_enum, :to_f, :to_i, :to_r, :to_s, :to_str, :to_sym, :tr, :tr!, :tr_s, :tr_s!, :trust, :undump, :unicode_normalize, :unicode_normalize!, :unicode_normalized?, :unpack, :unpack1, :untaint, :untrust, :untrusted?, :upcase, :upcase!, :upto, :valid_encoding?, :yield_self]

Many many thanks.

You would need reflection to do those things which most Scala developers (including myself) prefer not to use.

Why do you want to do that? What is the meta-problem you are trying to solve?

For instance, I want to know how many member methods exist for the String class, it’s better to get the members list in scala shell via a built-in API.


But for what? Like I don’t understand if that is you when programming or you want your program to dynamically pick a method?

If the first the IDE should show you, or you can press tab after: "foo". on the REPL and it will show you all the methods, you can also just check the Scaladoc & Javadoc.