{"id":11525,"date":"2015-06-02T17:46:42","date_gmt":"2015-06-02T17:46:42","guid":{"rendered":"https:\/\/www.happiestminds.com\/blogs\/?p=2539"},"modified":"2024-04-11T06:04:39","modified_gmt":"2024-04-11T06:04:39","slug":"spark-in-action-processing-data","status":"publish","type":"post","link":"https:\/\/www.happiestminds.com\/blogs\/spark-in-action-processing-data\/","title":{"rendered":"Spark In Action : Processing Data"},"content":{"rendered":"
<\/div>
\n

This blog is intended for data engineers, data scientists who are planning to use Spark & Scala, and for anyone who is interested in learning Spark trough practice.<\/p>\n

Understanding How to use Spark for Data Exploration<\/h2>\n

The goal of the blog is to provide a quick start to Apache Spark and how to go about its application<\/p>\n

    \n
  1. RDD<\/li>\n
  2. Transformations<\/li>\n
  3. Actions<\/li>\n
  4. Perform Joins<\/li>\n
  5. Spark SQL<\/li>\n
  6. Spark Automatic Json Schema Inference Capability<\/li>\n
  7. Caching<\/li>\n
  8. Lazy Evaluation<\/li>\n<\/ol>\n

    Quick Intro to Spark<\/h2>\n

    Apache Spark is often referred to as the ‘Swiss Knife’, because of its ability to do different tasks with ease. It can handle data cleansing, data exploration, feature extraction, sql query, machine learning, complex graph algorithms, building streaming applications, etc…<\/p>\n

    Getting Started with Spark<\/h2>\n

    In case you do not have a spark setup on your machine, do not worry. It takes less than 2 minutes to get started. Download spark from\u00a0“Apache Spark”. Extract it and start the Scala REPL from $Spark Home\/bin\/spark-shell.<\/p>\n

    Note:-\u00a0 $Spark_Home\u00a0<\/strong>is the location where you extracted.<\/p>\n

    Loading Data Set<\/h2>\n

    Download the required Data Set from\u00a0github<\/a>\u00a0repository and execute the below steps in the spark-shell.<\/p>\n

    val<\/strong>rawData = sc.textFile(“path\/usagov_bitly_data2012-03-16-1331923249.txt”<\/strong>)<\/p>\n

    rawData: org.apache.spark.rdd.RDD[String] = usagov_bitly_data2012-03-16-1331923249.txt MappedRDD[1] at textFile at\u00a0:12<\/p>\n

    We just created an RDD called raw data.<\/p>\n

    What is RDD :\u00a0<\/strong>In simplest words, it can be expressed as a data structure distributed across many machines. RDD is the core of Spark, and all operations revolve around it.<\/p>\n

    Spark provides two kinds of operations, one is transformation and another is action.<\/p>\n

    Transformation and Actions<\/h2>\n

    What is Transformation:\u00a0<\/strong>Transformation converts one RDD to another RDD.<\/p>\n

    What is Action:\u00a0<\/strong>Action on the other hand produces an output like printing the data and storing the data .<\/p>\n

    In this case, we created an RDD of type string.<\/p>\n

    What is sc:\u00a0<\/strong>For Spark shell , Spark creates a context\u00a0sc. It is the main entry point for Spark functionality. It helps you tcreate RDD, accumulate and broadcast variables.<\/p>\n

    What is Text File:<\/strong>\u00a0Text File method helps you read data from \u00a0local file system, Hadoop file system or any hadoop-supported system URI.<\/p>\n

    Before proceeding further, Let’s try to understand about the data set.<\/p>\n

    In 2011, URL shortening service bit.ly partnered with the United States government’s website – usa.gov to provide a feed of anonymous data gathered from users who shorten links ending with .gov or .mil. Each line in each file contains a common form of web data known as ‘JSON’ .<\/p>\n

    So one of the key points is, we are going to deal with JSON data, and each line in the file represents a JSON input.<\/p>\n

    Let’s try to see what our RDD contains. Spark provides methods like\u00a0collect,\u00a0take(number of records),\u00a0first\u00a0to access the elements in the RDD.<\/p>\n

    Collect()\u00a0<\/strong>: It brings the entire data to the driver program. In our case, driver program is Spark Shell. You cannot collect when the data set is huge, that it cannot fit into the memory of a single machine. You can use and collect the same when your dataset is limited.<\/p>\n

    Take(n)<\/strong>\u00a0: It complements the limitation of collect. It returns the\u00a0n\u00a0<\/strong>elements of the RDD.<\/p>\n

    First :<\/strong>\u00a0It returns only one element from the RDD. Useful when you want to quickly check whether your operations worked correctly.<\/p>\n

    Let’s try first on our rawData. “rawData.first”<\/p>\n

    res0: String = { “a”: “Mozilla\\\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\\\/535.11 (KHTML, like Gecko) Chrome\\\/17.0.963.78 Safari\\\/535.11”, “c”: “US”, “nk”: 1, “tz”: “America\\\/New_York”, “gr”: “MA”, “g”: “A6qOVH”, “h”: “wfLQtf”, “l”: “orofrog”, “al”: “en-US,en;q=0.8”, “hh”: “1.usa.gov”, “r”: “http:\\\/\\\/www.facebook.com\\\/l\\\/7AQEFzjSi\\\/1.usa.gov\\\/wfLQtf”, “u”: “http:\\\/\\\/www.ncbi.nlm.nih.gov\\\/pubmed\\\/22415991”, “t”: 1331923247, “hc”: 1331822918, “cy”: “Danvers”, “ll”: [ 42.576698, -70.954903 ] }<\/p>\n

    Spark SQL JSON Inference<\/h2>\n

    To understand more about the data set, we should bring JSON data to a structured form like table. For that, we need to extract the elements . There are several JSON libraries which can help in parsing data. We will use\u00a0Spark Sql\u00a0capability of inferring JSON Schema to structure our data.<\/p>\n

    import <\/strong>org.apache.spark.sql.SQLContext<\/p>\n

     <\/p>\n

    val<\/strong>sqlContext = new <\/strong>SQLContext(sc)<\/p>\n

    val<\/strong>recordsJson = sqlContext.jsonRDD(rawData)<\/p>\n

    recordsJson.registerTempTable(“records”<\/strong>)<\/p>\n

     <\/p>\n

    SparkSql is a subproject of Apache Spark \u00a0to deal with structured data. The beauty of it is that it works seamlessly with Spark programs. It provides various features to handle data from Hive, JSON , parquet \u00a0and thrift service for external programs to connect. There are many more features that Spark SQL supports.<\/p>\n

    Lets try to check what is the schema of our data. SparkSQL comes with a printSchema method.<\/p>\n

    recordsJson.printSchema<\/p>\n

    root<\/p>\n

    |– _heartbeat_: integer (nullable = true)<\/p>\n

    |– a: string (nullable = true)<\/p>\n

    |– al: string (nullable = true)<\/p>\n

    |– c: string (nullable = true)<\/p>\n

    |– cy: string (nullable = true)<\/p>\n

    |– g: string (nullable = true)<\/p>\n

    |– gr: string (nullable = true)<\/p>\n

    |– h: string (nullable = true)<\/p>\n

    |– hc: integer (nullable = true)<\/p>\n

    |– hh: string (nullable = true)<\/p>\n

    |– kw: string (nullable = true)<\/p>\n

    |– l: string (nullable = true)<\/p>\n

    |– ll: array (nullable = true)<\/p>\n

    |\u00a0\u00a0 |– element: double (containsNull = false)<\/p>\n

    |– nk: integer (nullable = true)<\/p>\n

    |– r: string (nullable = true)<\/p>\n

    |– t: integer (nullable = true)<\/p>\n

    |– tz: string (nullable = true)<\/p>\n

    |– u: string (nullable = true)
    \nOne advantage of SparkSQL is that we can use the spark functionalities also. Now let’s try to proceed with our exploration of the dataset. Let’s take one particular field say\u00a0‘tz‘\u00a0<\/strong>which stands for time zone. Lets quickly write a SQL to extract the data.<\/p>\n

    val<\/strong>\u00a0<\/strong>x =\u00a0sqlContext.sql(“select tz from records”<\/strong>)<\/p>\n

    Now, ‘x’ will contain only the time zone records.<\/p>\n

    Lets try to find how many times each time zone occurs in the data set<\/h2>\n

    How can we solve it. Let’s get each element, and then map it to 1 to form a key value pair.<\/p>\n

    x.map(row => row(0)).map(temp => (temp,1))<\/p>\n

    Here\u00a0x\u00a0is a Schema RDD.<\/p>\n

    map(f):\u00a0<\/strong>\u00a0map is a transformation applied on the RDD . It accepts a function as input, and applies it to each element of RDD in a parallel fashion.
    \n\"json\"<\/p>\n

    Let’s see how the elements have transformed . We will do\u00a0take(5)\u00a0<\/strong>on the above RDD .<\/p>\n

    (America\/New_York,1)<\/p>\n

    (America\/Denver,1)<\/p>\n

    (America\/New_York,1)<\/p>\n

    (America\/Sao_Paulo,1)<\/p>\n

    (America\/New_York,1)<\/p>\n

    Now, a simple question that you can be asking yourself could be ‘why I am pairing each field with 1?’. Basically we converted our fields to a tuple of key-value pairs.<\/p>\n

    Spark provides useful transformations on key-value pair RDD like reduceByKey ,groupByKey , countByKey and many more.<\/p>\n

    Let’s get back to our question,\u00a0<\/strong>how many times each time zone occurs .<\/strong>\u00a0Hope you would have guessed \u00a0it, let us use reduceByKey on the data.<\/p>\n

    x.map(row => row(0)).map(temp => (temp,1)).reduceByKey((x,y) =>x+y).take(5).foreach(println)<\/p>\n

    (null,120)<\/p>\n

    (Europe\/Brussels,4)<\/p>\n

    (Europe\/Vienna,6)<\/p>\n

    (Europe\/Athens,6)<\/p>\n

    (America\/Puerto_Rico,10)
    \nNote:<\/strong>\u00a0Remember that transformations in Spark are Lazy, it means till you perform an action like count ,first, collect, saveAsTextFile transformations are not evaluated.<\/p>\n

    Looking at the result , we can recognize that there are\u00a0null\u00a0values . Let’s try to remove records that contain\u00a0nullvalues. Spark comes with a handy\u00a0filter\u00a0method.\u00a0<\/strong>
    \nx.map(row => row(0)).filter(x => x!=null).map(temp => (temp,1)).reduceByKey((x,y) =>x+y).take(5).foreach(println)<\/p>\n

    (Europe\/Brussels,4)<\/p>\n

    (Asia\/Bangkok,6)<\/p>\n

    (Pacific\/Honolulu,36)<\/p>\n

    (America\/Santo_Domingo,1)<\/p>\n

    (Europe\/Bucharest,4)<\/p>\n

    Let’s see how we could sort the data. We will use\u00a0sortBy\u00a0.<\/p>\n

    x.map(row => row(0)).filter(x => x!=null).map(temp => (temp,1)).reduceByKey((x,y) =>x+y).sortBy(_._2,false).take(5).foreach(println)<\/p>\n

    (America\/New_York,1251)<\/p>\n

    (,521)<\/p>\n

    (America\/Chicago,400)<\/p>\n

    (America\/Los_Angeles,382)<\/p>\n

    (America\/Denver,191)<\/p>\n

    Looking at the data , we can guess that the data needs more cleansing, which I will leave to you.
    \nUse the filter() on the RDD and remove the\u00a0empty rows.<\/p>\n

    We can simplify the entire task by writing a\u00a0SQL\u00a0<\/strong>query on\u00a0records\u00a0<\/strong>table .<\/p>\n

    sqlContext.sql(“select tz,count(tz) as total from records where tz != ” and tz is not NULL group by tz order by total desc”).take(5).foreach(println)<\/p>\n

    [America\/New_York,1251]<\/p>\n

    [America\/Chicago,400]<\/p>\n

    [America\/Los_Angeles,382]<\/p>\n

    [America\/Denver,191]<\/p>\n

    [Europe\/London,74]<\/p>\n

    In the next blog we will look into how joins, caching, lazy evaluation works.<\/p>\n<\/div>\n

    \r\n
    \r\n \r\n <\/i>\r\n <\/a>\r\n 0 <\/span>\r\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"

    This blog is intended for data engineers, data scientists who are planning to use Spark & Scala, and for anyone who is interested in learning Spark trough practice. Understanding How to use Spark for Data Exploration The goal of the blog is to provide a quick start to Apache Spark and how to go about […]<\/p>\n","protected":false},"author":25,"featured_media":1313,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[225,312,218,123,661],"tags":[313,201,927,940,941,1506,1592,1593],"class_list":["post-11525","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-big-data","category-digital-transformation","category-technology","category-tools","tag-analytics","tag-big-data","tag-data-analytics","tag-data-science","tag-data-scientists","tag-scala","tag-spark","tag-spark-data-processing"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/posts\/11525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/comments?post=11525"}],"version-history":[{"count":1,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/posts\/11525\/revisions"}],"predecessor-version":[{"id":12282,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/posts\/11525\/revisions\/12282"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/media\/1313"}],"wp:attachment":[{"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/media?parent=11525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/categories?post=11525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.happiestminds.com\/blogs\/wp-json\/wp\/v2\/tags?post=11525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}