Data transform

The transformation module is responsible for making the data transformations.

SQL transformation

This module makes SQL expressions available to transform. Example:

from pyiris.ingestion.transform import SqlTransformation

sql_transformation = SqlTransformation(name='divide', 
                                       description='Unit price division', 
                                       to_column="unit_price", 
                                       sql_expression="price/quantity")

transformed_dataframe = sql_transformation.transform(dataframe=extracted_dataset)

Hash transformation

This module returns a hash transformation based on an inputted column. Example:

from pyiris.ingestion.transform import HashTransformation

hash_transformation = HashTransformation(name='Hash CPF', 
                                         description='Hash CPF to be according to LGPD', 
                                         from_columns=["cpf"])

transformed_dataframe = hash_transformation.transform(dataframe=extracted_dataset)

Custom transformation

This module gives for the user tools to customize the dataframe, with the main custom features. Example of uses:

from pyiris.ingestion.transform.transformations.custom.custom import divide
from pyiris.ingestion.transform.transformations.custom_transformation import CustomTransformation

custom_transformation = CustomTransformation(name='middle_price',
                                             description='Dividing two fictitious columns (price/quantity) to generate column middle_price',
                                             method=divide,
                                             to_column='middle_price', 
                                             column1='price', 
                                             column2='quantity')

transformed_dataframe = custom_transformation.transform(dataframe=extracted_dataset)

Transform Service

The class pyiris.ingestion.transform.TransformService works as a service. You can execute some transformations in sequence, or only one. Follow the example of uses:

from pyiris.ingestion.transform import TransformService, HashTransformation, SqlTransformation

transform_service = TransformService(
    transformations=[
        SqlTransformation(name='divide', 
                          description='Getting middle price', 
                          to_column="middle_price", 
                          sql_expression="price/quantity"),
        HashTransformation(name='Hash CPF', 
                           description='Hash CPF to be according to LGPD', 
                           from_columns=["seller_cpf"])
    ]

)
transformed_dataframe = transform_service.handler(dataframe=extracted_dataset)

To have more information, please, access the code docstring in Pyiris modules.