Models

The models module aims to standardize and guarantee users will always tie together all of the necessary definitions for their models training jobs.

BaseIrisModel

This object was designed for users to extend and declare their models objects. It is based on mlflow’s PythonModel and enforces that the user has to define the following by implementing abstract methods:

• load_context: How to load their serialized objects

• load_training_data: How to read the training data to be used

• fit: How to train the model

• predict: How to make the predictions

• get_signature: How to infer the signature for the trained models

A possible usage example is:

import cloudpickle
from mlflow.models import infer_signature
from sklearn.ensemble import RandomForestClassifier

from pyiris.infrastructure import Spark
from pyiris.ingestion.extract import FileReader
from pyiris.intelligence.models import BaseIrisModel


class MyCustomModel(BaseIrisModel):
    
    def __init__(self):
        self.estimator = None

    def load_context(self, context):
         with open(context.artifacts["estimator"], "rb") as f:
            self.estimator = cloudpickle.load(f)

    def load_training_data(self):
        iris_spark = Spark()
    
        dataframe = FileReader(
            table_id='clientes_features', 
            data_lake_zone='consumezone',
            country='Brazil', 
            path='Iris/Intelligence/ExampleAlgorithm/Features', 
            format='parquet'
        ).consume(spark=iris_spark)
        return dataframe

    def fit(self):
        dataframe = self.load_training_data()
        self.estimator = RandomForestClassifier()
        self.estimator.fit(dataframe)
        
        with open("fitted_model.pkl", "wb") as f:
            cloudpickle.dump(self.estimator, f)

    def predict(self, context, data):
        return self.estimator.predict(data)

    def get_signature(self):
        signature_dataframe = self.load_training_data()
        signature = infer_signature(signature_dataframe)
        return signature

Other implementations

All users are welcome to extend this class and create their own reproducible, standardized and customized model that can be used multiple times by anyone else in the team. The single requirement is that it must inherit the BaseIrisModel object.