Skip to content

Add an out-of-fold target encoder (OOFMeanEncoder) #1050

Open

Description

@hx-max

Is your feature request related to a problem? Please describe.

MeanEncoder replaces a category with the mean target observed for that category. When it is
estimated and applied on the same rows, every row contributes its own target to its own
encoding. With k rows per category each row is responsible for 1/k of its own encoded
value, and at k = 1 the encoding is the label itself:

rows per category weight of the row's own target
1 100%
3 33%
10 10%
1000 0.1%

So the model scores well in training and validation collapses. MeanEncoder's documentation
warns about this, but the library offers no encoder that avoids it, and the alternatives live
in other packages.

Describe the solution you'd like

A transformer that learns its mapping out of fold, so no row is encoded using its own target.
This is the same idea as category_encoders.TargetEncoder(cv=...) and CatBoost's ordered
target statistics, neither of which integrates with the feature-engine / scikit-learn pipeline
conventions used here.

Proposed API, following the conventions of the existing encoders:

from feature_engine.encoding import OOFMeanEncoder

enc = OOFMeanEncoder(
    variables=["cabin", "embarked"],
    cv=5,                 # number of folds
    smoothing=10,         # additive smoothing towards the prior
    unseen="encode",      # encode unseen categories with the prior
    shuffle=True,
    random_state=0,
    stratify=True,
)
X_train = enc.fit_transform(X_train, y_train)   # out of fold
X_test = enc.transform(X_test)                  # full-data mapping
X_train_orig = enc.inverse_transform(X_train)
  • fit_transform splits the training data into cv folds, learns the mapping for each fold
    from the remaining folds, and encodes the held-out fold with it.
  • fit learns the mapping from the full training set and stores it in encoder_dict_, with
    the prior in y_prior_, so transform behaves like MeanEncoder at inference time.
  • variables, missing_values, ignore_format, unseen, return_empty and smoothing
    work as they do elsewhere in encoding.

Describe alternatives you've considered

  • Tell users to wrap MeanEncoder in a KFold loop by hand. It works, but it is easy to get
    wrong and it is the reason the issue exists.
  • Recommend category_encoders or CatBoost. Both are good, but it means a second encoder
    library in a project that already depends on feature-engine, and category_encoders has a
    different API for variables / unseen categories / returning DataFrames.

Additional context

I have an implementation with 46 tests ready, and tests/test_encoding passes in full (389
tests) including test_check_estimator_encoders.py:

  • feature_engine/encoding/oof_mean_encoding.py
  • tests/test_encoding/test_oof_mean_encoder.py
  • docs/api_doc/encoding/OOFMeanEncoder.rst plus the encoder table and toctree entry

Happy to open the PR, or to wait if you would rather scope it differently — for example a
cv argument on MeanEncoder instead of a separate class. I am also glad to add a
docs/whats_new entry if you tell me the format you prefer for unreleased versions.

Activity

  1. hx-max commented on Oct 13, 2026

    @hx-max
    Author

    Hi @solegalli, I've opened PR #1102 with a complete implementation of the API sketched in this
    issue.

    What's in it:

    • feature_engine/encoding/oof_mean_encoding.py -- OOFMeanEncoder following the existing
      feature-engine conventions (variables, missing_values, ignore_format, unseen, return_empty,
      smoothing, plus cv / shuffle / random_state / stratify).
    • 46 tests in tests/test_encoding/test_oof_mean_encoder.py, covering the leakage property itself,
      all three unseen modes, stratify degradation on continuous targets, smoothing (int, float and
      'auto'), missing_values, ignore_format, return_empty, inverse_transform and feature-name
      handling.
    • Registered in tests/test_encoding/test_check_estimator_encoders.py so the class is covered by
      the project's own check_feature_engine_estimator.
    • An API docs page, plus the encoder comparison table and toctree entry.

    The branch is rebased on current main (d866312). Locally: tests/test_encoding passes in full
    (392 tests), flake8 is clean, the sphinx -W docs build has no warnings, and mypy reports no
    issues in the files this PR touches -- the one remaining mypy error is a pre-existing one in
    feature_engine/datetime/datetime_subtraction.py that reproduces on unmodified main.

    I see PR #1097 also references this issue. Happy for you to pick whichever implementation fits
    the roadmap, and glad to share the test suite if that helps consolidate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions