Introduction to Spark 2.0

Rklick Solutions LLC

Overview of Dataset , Dataframe and RDD API :

Resilient Distributed Datasets (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster.

But due to facing issue related to advanced optimization move to dataframe.

Dataframe brought custom memory management and runtime code generation which greatly improved performance. So in last year most of the improvements went into Dataframe API.

A DataFrame is a Dataset organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs.

Though dataframe API solved many issues, it was not a…

View original post 742 more words

Effective Programming In Scala – Part 3 : Powering Up your code implicitly in Scala

Discover type annotations versus ascriptions, adding methods to class in implicit way and monkey patching.


Hi Folks,
In this series we talk about the concepts that provide a better definition to the code written in scala. We provide the methods with some definitions that lead to perform a task in a better way. Lets have a look at what we have done in the series so far,

Effective Programming in Scala – Part 1 : Standardizing code in better way
Here we covered the better solution to code, so that it cannot lead the any kind of code styling or formatting errors behind. Provided the way to use scala properties and collections in a better way according to their behaviour.
Effective Programming In Scala – Part 2 : Monads as a way for abstract computations in Scala
Here we provided short hand solutions to the long and lengthy writing of code to perform some task in a predefined sequence. Provided the better use of comprehensions…

View original post 762 more words

Initializing data with eager binding in play


I just came across a scenario where I had not to choose Play GlobalSettings to execute some tasks on start up of the application, as it is being deprecated in Play 2.5. So as the alternative what we can do to make things happen at the time when application just gets started.

So whatever was done using GlobalSetting.onStart and GlobalSetting.beforeStart needs to be implemented using the dependency injection.

For this, we will just provide the eager binding for the class implementation that needs to perform some tasks just before the application actually gets started.

(Eager binding is used to start up a service when an application starts.)

Then provide those implementations in the constructor of the class for which we have provided eager binding.

So lets start going step by step to replace our onStart/beforeStart implementations with dependency injected class.

Firstly create a class say, MyDBModule which extends AbstractModule in which we will bind the class implementation that will…

View original post 248 more words

Integrate JWT with Akka HTTP


In this article we will discuss about , how to implement  authentication or authorization in Akka HTTP routes using JWT .

As we know Akka HTTP is full implementation of  server  and client side HTTP stack on top Akka actor and Akka stream . Now Let’s  we  talk about JWT.

what is JWT ?

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA.

What is the JSON Web Token structure?

JSON Web Tokens consist of three parts separated by dots “.”, which are:

  • Header
  • Payload
  • Signature

Let’s dive into  detail of each part.

Header :  The header typically…

View original post 214 more words

A basic application to handle multipart form data using akka-http with test cases in Scala


In my previous blogs, I have talked about file upload and its test cases using akka-http.

Now, in this blog I am going to explain the handling of multi part form data in akka-http with the help of a basic application which contains the code and its test cases.

So Let’s start with the dependencies for the application.
You have to add the following dependencies in you build.sbt :

Now, we have to create a handler to handle multipart form data :

Whenever a file will come in request, a new file will be created in temp directory of your system.

Below, you can find the test cases for the handler :

I hope, you enjoyed it and it will be helpful for you.

You can get full code here

Happy Blogging !!!

View original post

Dynamic Typing in Scala


Scala provides different ways to share common data between the files. In a simplified way, a class is often used to share the values by storing them in methods in that class. But those methods are statically typed. What we mean by statically typed ?

Here we mean by statically typed, is that a class having a method declared and used everywhere we want to assign information and fetch from it. Now if there is a requirement to have a unique method every time we store some information in and fetch.

As Scala provides the dynamic typing, we have the following profits,

  • Code written in Scala will be more understandable due to well formed method names and arguments,
  • Dynamic typing will create the method at the execution time, we need not to type it statically,
  • Need not to create many variables, methods to store or calculate some common calculations with…

View original post 519 more words

Introduction to Knockout js with Play template


Knockout is a JavaScript library that is based on Model–view–viewmodel (MVVM) that helps to build responsive UI display. This makes clear separation between model(Domain model/ stored data), view(User Interface) and view-model(presentation of model in which binder communicates between view and data-binder).

Why Knockout js?

1. Declarative bindings : This is a convenient way to bind the html to the data model. When we try to manipulates the DOM with the declarative binding all the bounded element remain connected

2.Dependency Tracking : Whenever you modify or update your data model, It will automatically updates the associated UI.

3.MVVM Design Pattern: knockout build a connection between ViewModel and View, and it any updation take place it will automatically update the UI element.

4.Extensible – This implements custom behaviors as new declarative bindings for easy reuse in just a few lines of code..

5. and many more


View original post 214 more words

Data ingestion from Google spreadsheet to Elasticsearch

Rklick Solutions LLC

In this blog we are elaborate how to ingest data from Google spreadsheet to Elasticsearch.

So, There are 5 steps to ingest data from Google spreadsheet to Elasticsearch. Please follow the below steps:

Step – 1)  Login to your account .

Step – 2) Open Spreadsheet and follow step.

Open the spreadsheet and click on Add one and type elasticsearch in search box.You would see below screen.


Now click to add elasticsearch plugin. After adding ,you have to give permission to it.After giving permission, elasticsearch plugin would be added into your account.

Step – 3) Add elasticsearch plugin :

–  Now click on  Add-ons , you would see below screen.


Step – 4) Fill Cluster Information :  

Click on send to cluster.Now  you would below screen


Here ,in right hand side ,you have to type Host and Port along with Username and Password.

Step -5) Test the…

View original post 126 more words