Data Cleansing: The Stage Prior to ETL Processes

[shukla7789]

Data cleansing is considered a prior and separate stage of ETL processes, which does not mean that its importance is less.
Currently, data cleansing is considered a prior and separate stage from the ETL processes themselves, which does NOT mean that its importance is less.

Importance of the cleaning stage
Ensures the quality of the data that we are going to process.
Avoid false or erroneous information.
Save disk space costs by eliminating duplicate information.
Speeds up queries due to the absence of duplicate or unusable data.
It helps to make correct strategic decisions.


ETL Processes: The Basis of Business Intelligence

Principles of the cleaning process
Apply data unification rules. For example, put the same amazon database letter in the row corresponding to sex, such as “M” for male and “F” for female. In this case, possible errors would also have to be identified or corrected, such as some user having put “M” as female.

Completeness validations. For example, checking that all customer data records of a bank contain the full postal address, triggering an alarm if any are missing.

Data standardization. The goal is to have all data of the same type entered in the same way. An example would be the DNI with the final letter of the tax identification number next to the numbers and without a hyphen separating them.



Data profiling
Although it is not yet considered an independent stage of the cleaning process, it is highly recommended to carry out a data profiling beforehand, where it is decided, through sampling, which changes to make and how to do them. In this way, we guarantee an optimal and completely standardized subsequent cleaning.