Over the last few years, you have invested heavily in building your Data Warehouse and Business Intelligence capabilities. And if you have done it right, you have been reaping the benefits of it in a meaningful way too. But we all know that it is handling only a reasonable amount of your structured data. Now we are in the Big Data era and you need to harness insights out of the large data sets – structured and un-structured – internal as well as external.
Apache Flink is the new star in the town. It is stealing the thunder from Apache Spark (at least in the streaming system) which has been creating buzz for some time now. This is because Spark streaming is built on top of RDDs which is essentially a collection, not a Stream. So now would be the right time to try your hands on Flink, even more so since Flink 1.0 was released last week.
Flink has two types of Windows – Tumbling Window and Sliding Window. The main difference between these windows is that Tumbling windows are non-overlapping whereas Sliding windows can beoverlapping.
In this article, I will try to explain these two windows and will also show how to write Scala program for each of these. Code used in this blog is also available in my Github Read more →
Spark RDDs are very simple at the same time very important concept in Apache Spark. Most of you might be knowing the full form of RDD, it is Resilient Distributed Datasets. Resilient because RDDs are immutable(can’t be modified once created), Distributed because it is distributed across cluster and Dataset because it holds data. Read more →
The Art of Story Telling in Data Science
I was watching a Television interview with one of the famous regional movie actors of India who rejected to play the lead role in a film that apparently turned out to be a big hit with another second in line actor. The reason he gave for his decision not to take up that film was – the director couldn’t articulate the story well so that he could visualize the final product. So it was not that the story was bad, it was not that his expertise on direction is limited, it was not the hero was not keen – but the main issue was the way in which the story was told.