Group by based features#

autofeats.features.group_by.categorical_statistics(df: Dataset) → List[Column][source]#

This function applies count_occurences_of_each_category and count_categorical_values to the dataset.

Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

autofeats.features.group_by.correlation_between_features(df: Dataset) → List[Column][source]#

This function generates the expressions used to calculate the correlation between numerical features. All numerical features will be combinated into groups of two.

Example of operation:

F.corr(F.col("x_1"), F.col("x_2")).alias("corr_between___x_1_x_2")
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

autofeats.features.group_by.count_categorical_values(df: Dataset) → List[Column][source]#

This function will apply count and countDistinct to the categorical columns as a whole.

Example of operation:

F.count(F.col("product_type"))
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

autofeats.features.group_by.count_occurences_of_each_category(df: Dataset) → List[Column][source]#

This function counts the occorurences of each category inside a categorical column.

Example of operation:

F.count(F.when(F.col("product_type") == "A", F.col("product_type")))
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

autofeats.features.group_by.get_categories_from_categorical_data(df: Dataset) → List[Dict[str, Any]][source]#

This function extracts categorical data from the categorical columns. This data is transformed in a dictionary with the column name as key and column values as values.

Example of return:

{"product_type": ["A", "B", "C"]}
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with information about columns categories.

Return type:

List[Dict[str, Any]]

autofeats.features.group_by.numerical_statistics(df: Dataset) → List[Column][source]#

This function generates numerical statisticst about the numerical columns in the dataset. The statistics will be calculated inside a time window, defined in the dataset instantiation.

Example of operation:

F.mean(F.col("paid_value")).alias("mean___paid_value")
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

autofeats.features.group_by.statistics_of_numerical_data_in_categorical_groups(df: Dataset) → List[Column][source]#

This function will calculate numerical statistics using a pivoted version of the dataset.

Example of operation:

F.mean(F.when(F.col("product_type") == "A", F.col("paid_value")))
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

List with the operations to apply in the dataframe

Return type:

List[Column]

Window based features#

autofeats.features.window.first_observation_value(df: Dataset) → DataFrame[source]#

This function will return the first row (observation) in the dataset. The columns selected will be only the numerical ones.

Example:

w = Window().partitionBy("customer_id").orderBy("buy_date")

df = df.withColumn("rn", F.row_number().over(w)).filter(F.col("rn") == 1).drop("rn")
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

Dataframe with the features

Return type:

DataFrame

autofeats.features.window.lags(df: Dataset, features: DataFrame, *args, **kwargs) → DataFrame[source]#

This function should be applied to the generated features. This function applies a lag function to the features table.

Example:

w = Window().partitionBy("customer_id").orderBy("date_ref")
df = df.withColumn("lag=1_mean___paid_value", F.lag("mean___paid_value").over(w))
Parameters:

  • df (Dataset) – Dataset initialized with necessary information

  • features (DataFrame) – Dataframe with features

Returns:

Dataframe with the features

Return type:

DataFrame

autofeats.features.window.last_observation_value(df: Dataset) → DataFrame[source]#

This function will return the last row (observation) in the dataset. The columns selected will be only the numerical ones.

Example:

w = Window().partitionBy("customer_id").orderBy(F.col("buy_date").desc())

df = df.withColumn("rn", F.row_number().over(w)).filter(F.col("rn") == 1).drop("rn")
Parameters:

df (Dataset) – Dataset initialized with necessary information

Returns:

Dataframe with the features

Return type:

DataFrame

autofeats.features.window.rate_between_actual_and_past_value(df: Dataset, features: DataFrame, *args, **kwargs) → DataFrame[source]#

This function should be applied to the generated features. This function generates the increase rate comparing the feature value in a date X with the feature value in a date X - 1.

Example:

w = Window().partitionBy("customer_id").orderBy("date_ref")
df = df.withColumn("increase_rate_mean___paid_value, (F.lag("paid_value").over(w) - F.col("paid_value"))/F.col("paid_value"))
Parameters:

  • df (Dataset) – Dataset initialized with necessary information

  • features (DataFrame) – Dataframe with features

Returns:

Dataframe with the features

Return type:

DataFrame