Data Engineering

What is ACID? Four “rules” ensure that a database transaction is timely. When a database adheres to these rules, it is said to be ACID-compliant. A for Atomicity: all database transactions can be broken down in smaller parts. Atomicity refers to the integrity of the whole transaction, not just one… 

What is a data lakehouse? For decades, two concurrent paradigms operated next to each other: the data lake and the data warehouse. The first one offering extremely cheap storage capacity for unstructured data formats, the second one offering easy querying capabilities on structured data formats. A data lakehouse is a… 

What is a data mart? A data mart is best regarded as a subset of a data warehouse. Oftentimes its use is oriented towards one specific business unit. They are useful to store summarized data, easily accessible to business users within that specific business unit. A data mart has several… 

Data monitoring is closely related to data testing. They both intend to preserve or improve the quality of data. But monitoring starts from another philosophy. Instead of testing data against known scenarios, monitoring your data means collecting, storing, and analyzing various properties of data. When a data monitoring system detects an anomaly,… 

The truth is that you rarely completely control how or what data is collected. That’s why you should evaluate your data for its quality. There are many dimensions to data quality. The list will be longer or shorter, depending on who you ask. Data validity: To store dates or times,… 

What is a data warehouse? A data warehouse is a central repository that contains all data of an organization. The data in a data warehouse often comes from a variety of data sources within marketing, sales, finance and operations. Most often, all data in the warehouse has already been cleaner…