JSON Lines is a format used in many locations on the web, and I recently came across the file format in Kaggle competition. The train data in this competition is JSONL format which contains the article_ID, time stamp and the event in each row from the e-commerce site Otto. …

TL:DR . The gist is one can use Intel i3 / 2GB RAM Linux Machine for “learning Data Engineering”, even complicated data like AirBnB datasets. The post is Learning log about my exploration and learning for past 4 months about Data Engineering My intention to write this post is to…

Individual Tools in any trade look trivial when we first use it. As we start mastering many tools, suddenly more problems get solved easily. Such is the case with the following 5 tools in Python. map filter reduce itertools.takewhile itertools.chain lambda function (anonymous function) Pyspark is about map => filter…

AWS EMR or Elastic Map Reduce is a cluster of EC2 nodes that can execute Distributed computing tasks. At the level of disk operations, and motherboards, the cluster is made up of servers with 2 TB (max) HDDs, 32GB RAMS and couple of 8 core processors running in Datacenters. Multiple…

I have been fascinated with data visualisation and analysis for long time. The more I learnt, I found there was more to learn and explore. ObservableHQ is online platform for showcasing the data visualisation and analysis skills. Exploring the platform, I found the community in observable is kind enough to…

That single picture is sufficient to tell all the services Cloud 9 can connect in AWS. That is because, AWS spawns a EC2 instance which in turn gives opportunity to work with everything that is on AWS or outstide AWS that can connect through the Network Interface. …

Partitioning Tables & AWS Glue: Learning Log 8 Someone forgot to tell that Data Engineering is really Data Base Engineering, or I did not read it in between the words. There is no escape from SQL queries any time soon. For most apps, atleast one SQL DB is there in…

Today there is Attention and then there is Action, we have forgotten the Interest and the Decision part of what Mitch Murry’s top motivator told to those sales guys in Glen Garry and Glen Ross. The Interest and Decision part is what Data Science and Data engineering automates. In…

There are variety of databases under the SQL and NOSQL umbrella. They come in all flavours and colored logos inside AWS, pretty much like the EC2 instances. Database itself is just a data-structure, which contains huge tables of records, arranged in rows(or columns). To operate on these data-structure, there are…