# Building training plans {{% notice style="note" title="This page is for plan authors" icon="user-gear" %}} Using the AI features needs nothing on this page. It documents how the platform team writes and packages a new training plan, and what the existing plans look like inside — useful when reading their metrics, their rules or their failure messages. {{% /notice %}} ## The framework Every training plan is a Python project built on the **training template framework**, the platform's fork of [MLflow Recipes](https://mlflow.org/docs/latest/recipes.html). The framework provides: - the **recipe engine** — the ingest → split → transform → train → evaluate → register pipeline, with anomaly detection recipes (`anomaly/v1@isolation_forest`, `anomaly/v1@autoencoder`) and a classification recipe (`classification/v1`) on top of MLflow's regression and classification ones; - a generic **inference server** — a FastAPI application that loads whatever model the plan produced and serves `POST /api/predict`, `GET /api/metrics` and `GET /health` over TLS; - the **`training-template` CLI** that runs the recipe, publishes the inference image and registers the result with the platform; - the **conventions** that let the Trainers API, the scheduler and the Inferencers API treat every plan alike. A plan is therefore mostly declarative: a `recipe.yaml`, a handful of Python functions, and templates. ## Layout of a plan repository ```text trainingplan-/ ├── recipe.yaml # the recipe: algorithm, steps, thresholds, metrics ├── model_schema.txt # JSON schema of a prediction request → inferencer.requestSchema ├── steps/ │ ├── ingest.py # load_file_as_cleaned_dataframe(path): read and clean the data │ ├── split.py # create_dataset_filter(df): rows to keep after the split │ ├── transform.py # transformer_fn(): the scikit-learn transformer to fit │ ├── train.py # estimator_fn(params): the unfitted estimator │ └── custom_metrics.py # metrics referenced from recipe.yaml ├── configurations/ │ ├── entrypoint.sh # the three CLI commands the container runs │ ├── config.json # how the inference server loads the model │ ├── predict_service.py # the prediction service class │ ├── schemas.py # pydantic RequestBody / ResponseBody │ ├── mapper.py # map_to_response_body(dict) → ResponseBody │ ├── requirements.txt # runtime dependencies of the inference image │ └── cli_config/rule_generator.py # generate_rule_creation_body() → the rule to create └── cli_templates/ ├── inference-docker-template.txt # Dockerfile of the inference image ├── rule_create_body.txt # template the rule generator fills in └── additional_trainer_dependencies.txt # extra RUN lines for the trainer image ``` ## The recipe ```yaml recipe: "anomaly/v1@isolation_forest" threshold: 0.95 # percentile of training scores that becomes the decision threshold steps: ingest: {{INGEST_CONFIG}} # filled in at run time from dataSourcePath split: split_ratios: [0.75, 0.125, 0.125] post_split_filter_method: create_dataset_filter transform: using: custom transformer_method: transformer_fn train: using: custom estimator_method: estimator_fn model_type: isolation-forest register: allow_non_validated_model: True ``` `recipe` selects the engine: `anomaly/v1@isolation_forest`, `anomaly/v1@autoencoder` or `classification/v1`. Each step names the function in `steps/` that customises it; `estimator_params` under `train` is passed to `estimator_fn`. `evaluate.validation_criteria` and `primary_metric` decide whether a model is *validated*; `register.allow_non_validated_model: True` registers it regardless, which is what the current plans do. Custom metrics are declared under `custom_metrics` and implemented in `steps/custom_metrics.py`. The `{{INGEST_CONFIG}}` placeholder is rendered when the container starts: the framework writes a run profile pointing the ingest step at `steps/ingest.py::load_file_as_cleaned_dataframe` with the location in `dataSourcePath`. That function is where a plan cleans its data and **enforces `minDataToTrain`**, failing the run with a clear message when there is not enough. ## The inference service After training, the framework copies its FastAPI server, the model artefacts and the plan's `configurations/` into a `dist/` folder and builds an image from `cli_templates/inference-docker-template.txt`. At start-up the server reads `configurations/config.json`: ```json { "service_class_name": "AnomalyPredictionService", "service_file_name": "predict_service.py", "model_path": "model/model.pkl", "transformers_path": "transformer.pkl" } ``` and loads the class from `predict_service.py`. The framework adds two keys after training — `threshold`, the calculated decision threshold, and `training_max_score` — so the running service knows the numbers its version was trained with. The class contract: ```python class AnomalyPredictionService: def __init__(self, model, threshold, transformer=None, max_score=None): ... @staticmethod def load_model(model_path: str): ... # joblib, Keras, torch — the plan decides def predict(self, data: dict) -> dict: ... # one request in, one result out ``` `schemas.py` declares the request and response as pydantic models — the request is what `model_schema.txt` describes in JSON schema, and what the inferencer publishes as `requestSchema` — and `mapper.py` turns the service's result into the response. Requests that fail the schema get `422`. ## The rule template `configurations/cli_config/rule_generator.py` must expose `generate_rule_creation_body()` returning the JSON of a rule creation request. The existing plans render `cli_templates/rule_create_body.txt` with the model name, the inferencer's service name and port, and plan configuration such as the APN. The rule is created **inactive**, in `default_channel`, named after the model, and only when no rule of that name exists there already. Keep the inferencer endpoint in a rule **parameter**, as the templates do. ## What a trainer job looks like from inside The container's `entrypoint.sh` runs three commands and forwards `SIGTERM` to whichever is running, so a job killed by its timeout still reports `TIMEOUT`: ```bash training-template generate-local-yaml --model-type isolation-forest training-template run --profile local training-template publish-inference isolation-forest ``` | Command | Does | |---|---| | `generate-local-yaml` | Writes `profiles/local.yaml`: MLflow experiment `-`, registered model `--`, tracking URI, artefact location and `dataSourcePath` | | `run --profile local` | Executes the recipe, logging parameters, metrics and the model to MLflow. A failure sends an `ERROR` callback and stops | | `publish-inference` | Downloads the latest run's artefacts into `dist/`, renders the inference Dockerfile, builds and pushes the image with a Kaniko job as `/-:v`, then creates the inferencer — with the rule, if `createRule` is `true` and the inferencer is new — or adds the image to the existing one, and sends the `OK` callback | The environment the job receives: | Variable | From | Meaning | |---|---|---| | `organizationId`, `inferencersAPICredential` | The per-trainer secret | Who the job acts as: the organization and the API key of the user who created the trainer | | `modelName`, `createRule`, `minDataToTrain`, `dataSourcePath` | The Trainers API | The model to produce, whether to create the rule, the plan's minimum, where the data is under `/data` | | The plan's `configFields` (`apn`…) | The trainer's `configuration` | Plan-specific settings | | `callbackUri` | The scheduler | Where to `POST` the final `OK` / `ERROR` / `TIMEOUT` | | `mlflowApiURL`, `mlflowArtifactsLocation`, `pvcName`, `imageRepoUrl`, `imageRepoUser`, `imageRepoPassword`, `inferencersAPI`, `inferencerServicePort`, `rulesAPI`, `serviceTlsCert`, `serviceTlsKey`, `kanikoTimeout`, `kanikoMode`, `kanikoRetries`, `trainerLogLevel`, `inferenceLogLevel`, `shouldShowAnomalyReason` | The platform's shared AI secret | Where MLflow, the registry, the Inferencers and Rules APIs live; the TLS material the inference server uses; build settings | ## Packaging a plan 1. **Build the base trainer image** — the framework plus the plan's extra dependencies from `cli_templates/additional_trainer_dependencies.txt` (TensorFlow for the autoencoder, PyTorch for the image plan): ```bash training-template build-base-trainer --image-name /base-trainer- --tag ``` 2. **Generate the trainer Dockerfile**, from the plan's directory, and build it from the parent directory: ```bash training-template generate-trainer-dockerfile --image-name /base-trainer- --tag cd .. && docker build -t /trainingplan-: -f trainingplan-/generated/Dockerfile . docker push /trainingplan-: ``` The generated Dockerfile copies `steps/`, `configurations/`, `recipe.yaml`, `model_schema.txt` and `cli_templates/` and sets `configurations/entrypoint.sh` as the entry point. 3. **Register the plan in the catalogue** with its `image.name` and `image.tag`, `configFields`, `columnData`, `minDataToTrain`, and the `source` types it supports. The catalogue is the platform's own collection; the [Training Plans API](../training_plans/) reads it, it does not write it. To try a plan without the platform, set the variables of the table above in a shell, run the three entrypoint commands by hand with a local MLflow, and start the built inference server with `training-template run-sbox-inference`. A `generate-mock-trainer` command produces an image that skips the training and only exercises the callbacks, the Inferencers API and the rule creation — the way the end-to-end tests check the loop without waiting for a real training. ## The AI platform itself The services this chapter documents — the Training Plans, Trainers and Inferencers APIs, the AI console, the MLflow tracking server and the shared AI secret — are versioned and deployed together by the platform team, alongside the scheduler and the file connector that belong to the core platform. Each API generates its server from the OpenAPI specification shown at the bottom of its page, so the specification is the contract, not a description of it.