Continuous Deployment Pipeline of your Deep Learning Model to the Cloud using AWS, S3 and CloudFormation — Introduction MLOps becomes more and more important. According to articles like this, “87% of data science projects never make it into production”. This is alarming. That means, that, on average, only 13 out of 100 models make it into production. The remaining 87 are just lost. They are stuck somewhere in…

Let’s say you run a large online bookstore. It’s open 24/7. Users may place orders and pay for them literally every minute or second. That means your website must quickly process lots of transactions involving small amounts of data like order ID and details, user ID, or credit card data…

Enterprise organisations are utilising cloud services to build data lakes, warehouses and automated ETL pipelines. In AWS Cloud, data lakes are built on top of Amazon S3 due to its durability, availability, scalability and cheap of cost. Amazon Athena is one of the best tools to query data from S3…

This time I am going to try to explain how can we use Apache Arrow in conjunction with Apache Spark and Python. First, let me share some basic concepts about this open source project. Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar…

PART I: Building the infrastructure — Data extractions are a very common use case in data teams. It consists of retrieving small data amounts out of data sources for a specific purpose. To see how this use case can be automated using AWS, we will go through a real pipeline I had to work on, to…

In this post, we will perform ETL operations using PySpark. We use two types of sources, MySQL as a database and CSV file as a filesystem, We divided the code into 3 major parts- 1. Extract 2. Transform 3. Load. We have a total of 3 data sources- Two Tables…

Quickly run python scripts inside a Docker image — We can create a Docker image in two main ways. First, using a Docker container, and second, using a Dockerfile. In this post, we will learn how to create a Docker image from a docker container. For this method, we are going to use the following Docker command. docker commit …

Ever wondered how Zomato knows what’s you favorite restaurant? Or how Myntra knows what dresses you like? They do this using your historical data. In fact Uber saves data every millisecond(as I was told in one of the interviews). All these companies leverage data to provide a better customer experience. …

Let’s face it… When it comes to running serverless applications, AWS Lambda (λ) service is one of the most easy to develop and deploy. …

Sometime we need to read data from S3 location. The data can be store in multiple file in S3. We can read data from multiple file like CSV or JSON and merge or create separate DataFrame for analysis. Here we will create Lambda function which read data from S3 location…

Introduction The goal of this article is to build a simple CI/CD pipeline using AWS CDK. In order to fully understand the steps, a basic understanding of the following: Docker AWS CodePipeline AWS CodeBuild AWS ECR AWS CDK We are going to build a simple Apache server using docker, and then…

As some of you might know that there is size limit to the package that we can deploy to AWS lambda. Zip format limit is 50 MB and if not zipped then 250 MB. …

Using Apache Airflow, Docker Today we will cover Apache Airflow installation. We will install Airflow on Docker. During this tutorial we will build a custom docker image of Airflow with additional libraries such as SQL Server provider since we have covered SQL Server as a source in the previous Airflow…

Photo by Gabriel Heinzer on Unsplash In the process of Data gathering a Data Analyst have to handle various forms of data from different sources. If you want to fetch data from some APIs that return in JSON format. JSON act as a Data structure to store that data. …

A beginner’s guide on reading API data using Python. — One of the most used capabilities today in the world of data analysis is to read and process data from an API, mainly because it is very common for access to data from many applications and services to be exposed using this interface. …

Background In this blog post, I will walk us through on how to create an ETL job using the following AWS Cloud Services: Amazon S3 (for intermediary data storage) AWS Glue Studio (main engine of the ETL job) Amazon Athena (query tool) ETL stands for Extract, Transform, and Load. In almost…

In the big data world, the biggest problem for many companies might be getting insights from data before it’s outdated. …

Recently I have been working on daoitright.xyz, a data center of DAOs with a focus on quantifying the fairness of the DAO governance. To keep the data up-to-date, I managed to build a data pipeline using AWS. …

Recently I had to load data from Google Cloud Storage into BigQuery and automate the process so that it will run each day at 8am. In this post I will cover the script I wrote and the process of setting up a Google Cloud Function and Cloud Scheduler. First, we…

This article will store a large amount of data in the AWS S3 bucket and use AWS glue to store the metadata for this data. And then, finally, use AWS Athena to query this large data using standard SQL. Note: If you find a typo in this article read through…