Data Transform¶

The transformation module is responsible for making the data transformations.

SQL transformation¶

This module makes SQL expressions available to transform. Example:

from pyiris.ingestion.transform import SqlTransformation

sql_transformation = SqlTransformation(name='divide', description='Unit price division', to_column="unit_price", sql_expression="price/quantity")

transformed_dataframe = sql_transformation.transform(dataframe=extracted_dataset)

Hash transformation¶

This module returns a hash transformation based on an inputted column. Example:

from pyiris.ingestion.transform import HashTransformation

hash_transformation = HashTransformation(name='Hash CPF', description='Hash CPF to be according to LGPD', from_columns=["cpf"])

transformed_dataframe = hash_transformation.transform(dataframe=extracted_dataset)

Custom transformation¶

This module gives for the user tools to customize the dataframe, with the main custom features. Example of uses:

from pyiris.ingestion.transform.transformations.custom.custom import divide
from pyiris.ingestion.transform.transformations.custom_transformation import CustomTransformation

custom_transformation = CustomTransformation(name='middle_price', description='Dividing two fictitious columns (price/quantity) to generate column middle_price', method=divide, to_column='middle_price', column1='price', column2='quantity')

transformed_dataframe = custom_transformation.transform(dataframe=extracted_dataset)

Custom transformation - snakecase_column_names¶

This method intends to rename all columns of a given dataframe to snake case.

The transformations applied are:

replacing letters containing accents and special characters (e.g. replacing ‘á’, ‘à’, ‘ã’ or ‘â’ to ‘a’);
replacing uppercase letters with underscore and lowercase;
removing duplicated undescore;
removing leading and trailing undescore;
removing all characters that are not allowed (a-z0-9_)

Example code¶

from pyiris.ingestion.transform.transformations.custom.custom import snakecase_column_names
from pyiris.ingestion.transform.transformations.custom_transformation import CustomTransformation

custom_transformation = CustomTransformation(name="snakecase_column_names", description="rename columns to snake case", method=snakecase_column_names)
transformed_dataframe = custom_transformation.transform(dataframe=extracted_dataset)

Example outputs¶

already_snake_case_column_name → already_snake_case_column_name notSNAKECaseColumnNameOne → not_snake_case_column_name_one NÕTSnákêCãsèColùmnNãmêTWÕ → not_snake_case_column_name_two

Transform Service¶

The class pyiris.ingestion.transform.TransformService works as a service. You can execute some transformations in sequence, or only one. Follow the example of uses:

from pyiris.ingestion.transform import TransformService, HashTransformation, SqlTransformation

transform_service = TransformService(
    transformations=[
        SqlTransformation(name='divide', description='Getting middle price', to_column="middle_price", sql_expression="price/quantity"),
        HashTransformation(name='Hash CPF', description='Hash CPF to be according to LGPD', from_columns=["seller_cpf"])
    ]

)
transformed_dataframe = transform_service.handler(dataframe=extracted_dataset)

To have more information, please, access the code docstring in Pyiris modules.