AWS Technology
AWS Glue and Databricks
AWS Glue and Databricks Unity Catalog are both data management tools, but they have some key differences in focus and functionality: AWS Glue Focus: ETL (Extra…
Read More »
Cloud Design GoalsImproving data queries, pipelines
On real world projects and deployments, you hear the lament that a datawarehouse or data engine ‘does not work’. Query response times are slow, it …
Read More »
AgileDataLake and its Challenges
DataLake The entire concept of a Data Operations Platform rests on top of a Data Lake. There is no simple definition of a Data Lake, but based on the author’s …
Read More »
Cloud Design GoalsParquet file format for Data Lakes
Parquet is a file format standard used in many enterprises. It allows the standardisation of files and provides a common framework for queries and storage. Par…
Read More »
Cloud Design GoalsData products – a summary
Data products are the end result of file or data movements to the cloud; ETL; processing; de-duplication; curation and storage in a consumable layer. There is …
Read More »
Cloud Design GoalsUnderstanding Data Lakes and Data Warehouses
In simple terms we can identify the differences between Data Lakes and Data Warehouses. Data Lake: A data lake is a centralized repository, usually a platform,…
Read More »
Application TransformationDigital and Data Transformation (Why)
Digital Transformation Digital transformation not a magic solution nor a buffet of word salads. DT is roughly defined as the integration of digital technologie…
Read More »
Cloud Design GoalsAWS S3 and Snowflake – use case summary
Both platforms are valid and will likely work together in larger enterprises. The tricky part is always access, entitlements and RBAC or TBAC. Across the 2 pla…
Read More »
AWS TechnologyAWS Sagemaker vs Databricks
A comparison of AWS Sage Maker and Databricks. Both satisify different use cases. A key aspect is the principle of ‘cloud native’, meaning that if …
Read More »