This page is for plan authors

Using the AI features needs nothing on this page. It documents how the platform team writes and packages a new training plan, and what the existing plans look like inside — useful when reading their metrics, their rules or their failure messages.

The framework

Every training plan is a Python project built on the training template framework, the platform’s fork of MLflow Recipes. The framework provides:

  • the recipe engine — the ingest → split → transform → train → evaluate → register pipeline, with anomaly detection recipes (anomaly/v1@isolation_forest, anomaly/v1@autoencoder) and a classification recipe (classification/v1) on top of MLflow’s regression and classification ones;
  • a generic inference server — a FastAPI application that loads whatever model the plan produced and serves POST /api/predict, GET /api/metrics and GET /health over TLS;
  • the training-template CLI that runs the recipe, publishes the inference image and registers the result with the platform;
  • the conventions that let the Trainers API, the scheduler and the Inferencers API treat every plan alike.

A plan is therefore mostly declarative: a recipe.yaml, a handful of Python functions, and templates.

Layout of a plan repository

trainingplan-<name>/
├── recipe.yaml                      # the recipe: algorithm, steps, thresholds, metrics
├── model_schema.txt                 # JSON schema of a prediction request → inferencer.requestSchema
├── steps/
│   ├── ingest.py                    # load_file_as_cleaned_dataframe(path): read and clean the data
│   ├── split.py                     # create_dataset_filter(df): rows to keep after the split
│   ├── transform.py                 # transformer_fn(): the scikit-learn transformer to fit
│   ├── train.py                     # estimator_fn(params): the unfitted estimator
│   └── custom_metrics.py            # metrics referenced from recipe.yaml
├── configurations/
│   ├── entrypoint.sh                # the three CLI commands the container runs
│   ├── config.json                  # how the inference server loads the model
│   ├── predict_service.py           # the prediction service class
│   ├── schemas.py                   # pydantic RequestBody / ResponseBody
│   ├── mapper.py                    # map_to_response_body(dict) → ResponseBody
│   ├── requirements.txt             # runtime dependencies of the inference image
│   └── cli_config/rule_generator.py # generate_rule_creation_body() → the rule to create
└── cli_templates/
    ├── inference-docker-template.txt          # Dockerfile of the inference image
    ├── rule_create_body.txt                   # template the rule generator fills in
    └── additional_trainer_dependencies.txt    # extra RUN lines for the trainer image

The recipe

recipe: "anomaly/v1@isolation_forest"
threshold: 0.95                         # percentile of training scores that becomes the decision threshold
steps:
  ingest: {{INGEST_CONFIG}}             # filled in at run time from dataSourcePath
  split:
    split_ratios: [0.75, 0.125, 0.125]
    post_split_filter_method: create_dataset_filter
  transform:
    using: custom
    transformer_method: transformer_fn
  train:
    using: custom
    estimator_method: estimator_fn
    model_type: isolation-forest
  register:
    allow_non_validated_model: True

recipe selects the engine: anomaly/v1@isolation_forest, anomaly/v1@autoencoder or classification/v1. Each step names the function in steps/ that customises it; estimator_params under train is passed to estimator_fn. evaluate.validation_criteria and primary_metric decide whether a model is validated; register.allow_non_validated_model: True registers it regardless, which is what the current plans do. Custom metrics are declared under custom_metrics and implemented in steps/custom_metrics.py.

The {{INGEST_CONFIG}} placeholder is rendered when the container starts: the framework writes a run profile pointing the ingest step at steps/ingest.py::load_file_as_cleaned_dataframe with the location in dataSourcePath. That function is where a plan cleans its data and enforces minDataToTrain, failing the run with a clear message when there is not enough.

The inference service

After training, the framework copies its FastAPI server, the model artefacts and the plan’s configurations/ into a dist/ folder and builds an image from cli_templates/inference-docker-template.txt. At start-up the server reads configurations/config.json:

{
  "service_class_name": "AnomalyPredictionService",
  "service_file_name": "predict_service.py",
  "model_path": "model/model.pkl",
  "transformers_path": "transformer.pkl"
}

and loads the class from predict_service.py. The framework adds two keys after training — threshold, the calculated decision threshold, and training_max_score — so the running service knows the numbers its version was trained with. The class contract:

class AnomalyPredictionService:
    def __init__(self, model, threshold, transformer=None, max_score=None): ...
    @staticmethod
    def load_model(model_path: str): ...          # joblib, Keras, torch — the plan decides
    def predict(self, data: dict) -> dict: ...    # one request in, one result out

schemas.py declares the request and response as pydantic models — the request is what model_schema.txt describes in JSON schema, and what the inferencer publishes as requestSchema — and mapper.py turns the service’s result into the response. Requests that fail the schema get 422.

The rule template

configurations/cli_config/rule_generator.py must expose generate_rule_creation_body() returning the JSON of a rule creation request. The existing plans render cli_templates/rule_create_body.txt with the model name, the inferencer’s service name and port, and plan configuration such as the APN. The rule is created inactive, in default_channel, named after the model, and only when no rule of that name exists there already. Keep the inferencer endpoint in a rule parameter, as the templates do.

What a trainer job looks like from inside

The container’s entrypoint.sh runs three commands and forwards SIGTERM to whichever is running, so a job killed by its timeout still reports TIMEOUT:

training-template generate-local-yaml --model-type isolation-forest
training-template run --profile local
training-template publish-inference isolation-forest
Command Does
generate-local-yaml Writes profiles/local.yaml: MLflow experiment <organizationId>-<modelName>, registered model <organizationId>-<modelName>-<model type>, tracking URI, artefact location and dataSourcePath
run --profile local Executes the recipe, logging parameters, metrics and the model to MLflow. A failure sends an ERROR callback and stops
publish-inference Downloads the latest run’s artefacts into dist/, renders the inference Dockerfile, builds and pushes the image with a Kaniko job as <imageRepoUrl>/<organizationId>-<modelName>:v<model version>, then creates the inferencer — with the rule, if createRule is true and the inferencer is new — or adds the image to the existing one, and sends the OK callback

The environment the job receives:

Variable From Meaning
organizationId, inferencersAPICredential The per-trainer secret Who the job acts as: the organization and the API key of the user who created the trainer
modelName, createRule, minDataToTrain, dataSourcePath The Trainers API The model to produce, whether to create the rule, the plan’s minimum, where the data is under /data
The plan’s configFields (apn…) The trainer’s configuration Plan-specific settings
callbackUri The scheduler Where to POST the final OK / ERROR / TIMEOUT
mlflowApiURL, mlflowArtifactsLocation, pvcName, imageRepoUrl, imageRepoUser, imageRepoPassword, inferencersAPI, inferencerServicePort, rulesAPI, serviceTlsCert, serviceTlsKey, kanikoTimeout, kanikoMode, kanikoRetries, trainerLogLevel, inferenceLogLevel, shouldShowAnomalyReason The platform’s shared AI secret Where MLflow, the registry, the Inferencers and Rules APIs live; the TLS material the inference server uses; build settings

Packaging a plan

  1. Build the base trainer image — the framework plus the plan’s extra dependencies from cli_templates/additional_trainer_dependencies.txt (TensorFlow for the autoencoder, PyTorch for the image plan):

    training-template build-base-trainer --image-name <registry>/base-trainer-<type> --tag <tag>

  2. Generate the trainer Dockerfile, from the plan’s directory, and build it from the parent directory:

    training-template generate-trainer-dockerfile --image-name <registry>/base-trainer-<type> --tag <tag>
cd .. && docker build -t <registry>/trainingplan-<name>:<version> -f trainingplan-<name>/generated/Dockerfile .
docker push <registry>/trainingplan-<name>:<version>

    The generated Dockerfile copies steps/, configurations/, recipe.yaml, model_schema.txt and cli_templates/ and sets configurations/entrypoint.sh as the entry point.

  3. Register the plan in the catalogue with its image.name and image.tag, configFields, columnData, minDataToTrain, and the source types it supports. The catalogue is the platform’s own collection; the Training Plans API reads it, it does not write it.

To try a plan without the platform, set the variables of the table above in a shell, run the three entrypoint commands by hand with a local MLflow, and start the built inference server with training-template run-sbox-inference. A generate-mock-trainer command produces an image that skips the training and only exercises the callbacks, the Inferencers API and the rule creation — the way the end-to-end tests check the loop without waiting for a real training.

The AI platform itself

The services this chapter documents — the Training Plans, Trainers and Inferencers APIs, the AI console, the MLflow tracking server and the shared AI secret — are versioned and deployed together by the platform team, alongside the scheduler and the file connector that belong to the core platform. Each API generates its server from the OpenAPI specification shown at the bottom of its page, so the specification is the contract, not a description of it.