Data Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
AWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
Standard Technology Stack for an AWS based Data Lake
A typical Technology Stack for a Data Lake. S3 as the Golden Source. Snowflake as a corporate Data Share with SQL use cases. If AWS-S3 and Redshift are not pro…
Read More »
Data Lake Basics and AWS
A data lake is a centralized repository that allows a firm to store structured and unstructured data at any scale. You can store your data as-is, without havin…
Read More »
AWS, Snowpipe, Snowflake
Moving data from S3 to Snowflake to satisfy use cases around analysis, corporate reporting, or cross-domain information collaboration is best achieved through …
Read More »
Data Glossary, Dictionary, Catalogue
The Data Glossary focuses on business terminology and definitions, bridging the gap between business and IT. A data glossary is a document that defines and des…
Read More »
AWS Sagemaker vs Databricks
A comparison of AWS Sage Maker and Databricks. Both satisify different use cases. A key aspect is the principle of ‘cloud native’, meaning that if …
Read More »
SaaS design best practices
1. Scalability: The Backbone of SaaS Architecture Scalability is at the heart of SaaS development. As the user base grows, a SaaS application must be able to h…
Read More »