> ## Documentation Index
> Fetch the complete documentation index at: https://pioneer-kelton-add-decoder-inference-prices.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Pioneer training jobs: lifecycle, metrics, and weights
> Understand how Pioneer training jobs work — from submitting a job and polling status to reading metrics, stopping jobs, and downloading trained model weights.
Fine-tuning in Pioneer adapts a base model to your specific task and domain using your labeled dataset. You submit a training job through the API, Pioneer handles the compute, and you get back a trained model you can call for inference or download. The whole process is asynchronous — you start the job, then poll until it finishes.
Pioneer uses supervised fine-tuning (`sft`) for all new training jobs. See the [LLM fine-tuning guide](/guides/fine-tune-llm) for dataset formatting and decoder training examples.
## Training job lifecycle
A training job's `status` field moves through several states. The main path is:
Your job has been accepted and is queued for execution. Pioneer is allocating compute.
Training is actively executing on the provider.
GPU training finished successfully. Loss metrics are available on the job record (see [Polling status and reading metrics](#polling-status-and-reading-metrics)), and checkpoints are ready to download or deploy.
Intermediate post-training steps — Pioneer is normalizing and packaging the trained artifact. You'll typically only see these transiently between `complete` and `deployed`.
The trained adapter is live on an inference provider and ready to serve requests via `model_id`.
A job can also end in **`errored`** (an error occurred during training), **`stopped`** (you gracefully halted it with `POST /felix/training-jobs/:id/stop` — checkpoints are preserved), **`terminated`** (you called `POST /felix/training-jobs/:id/terminate`, which stops the job *and* permanently deletes its checkpoints — irreversible), or **`paused`**.
## Key parameters
| Parameter | Required | Description |
| --------------- | -------- | -------------------------------------------------------------------------------------------------------- |
| `model_name` | Yes | A name for your trained model, used to identify it in your account. |
| `base_model` | Yes | The model ID to fine-tune. Use a value from `GET /base-models` or a checkpoint UUID from a previous job. |
| `datasets` | Yes | An array of dataset objects: `[{"name": "your-dataset-name"}]`. |
| `training_type` | No | `"lora"` (default, parameter-efficient) or `"full"` (all weights). Decoder LLM training is LoRA-only. |
| `nr_epochs` | No | Number of training epochs. Defaults to 100, except decoder base models default to 10 when omitted. |
| `learning_rate` | No | Learning rate. Omit to use the default for the chosen base model. |
`base_model` is required and must match a model-ID or UUID shape — not a free-form string. Omitting it, or sending a malformed value, returns `422`. A well-formed value that doesn't match any model available for training returns `400` instead.
## Supported training targets
New training jobs support only the following target families:
| Family | Supported targets |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nemotron 3.5 Lightning decoders | `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`, `fastino/Fastino-Nemotron-3.5-Lightning-Finance`, `fastino/Fastino-Nemotron-3.5-Lightning-Healthcare` |
| GLiNER encoders | GLiNER2 Base, Large, Multi, and Multi Large |
Use `GET /base-models?supports_training=true` immediately before creating a
job. It is the live source of truth for target availability.
## Starting a training job
```bash theme={null}
curl -X POST https://api.pioneer.ai/felix/training-jobs \
-H "X-API-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model_name": "my-ner-model",
"base_model": "fastino/gliner2-base-v1",
"datasets": [{"name": "my-ner-dataset"}],
"training_type": "lora",
"nr_epochs": 5,
"learning_rate": 5e-5
}'
```
The response returns the full job record immediately, including a UUID `id` and initial status:
```json theme={null}
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"model_name": "my-ner-model",
"base_model": "fastino/gliner2-base-v1",
"status": "requested",
"nr_epochs": 5,
"learning_rate": 5e-5
}
```
Save the `id` — you'll use it to poll status, retrieve metrics, and run inference against your trained model.
## Polling status and reading metrics
Poll the job endpoint until `status` reaches a terminal value — `complete`, `deployed`, `errored`, `stopped`, or `terminated`:
```bash theme={null}
curl https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6 \
-H "X-API-Key: YOUR_API_KEY"
```
The `metrics` field always includes loss values once training starts, plus F1/precision/recall/accuracy if a separate evaluation has been run against the resulting model:
```json theme={null}
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "complete",
"metrics": {
"final_training_loss": 0.12,
"final_validation_loss": 0.18,
"best_validation_loss": 0.15,
"eval_f1_score": 0.94,
"eval_precision": 0.96,
"eval_recall": 0.92,
"eval_accuracy": 0.95
}
}
```
To retrieve structured stdout/stderr log lines for the job:
```bash theme={null}
curl https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/logs \
-H "X-API-Key: YOUR_API_KEY"
```
This returns a JSON list of log entries (`{id, timestamp, level, message, source}`) — it's a point-in-time fetch, not a live stream. Poll it periodically while the job is `running` to follow progress.
## Stopping or terminating a job
To gracefully halt a running job while preserving its checkpoints:
```bash theme={null}
curl -X POST https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/stop \
-H "X-API-Key: YOUR_API_KEY"
```
The job status changes to `stopped`. Checkpoints saved before the stop remain available for deployment or download.
To permanently end a job and delete its checkpoints instead, use `/terminate`:
```bash theme={null}
curl -X POST https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/terminate \
-H "X-API-Key: YOUR_API_KEY"
```
`/terminate` stops the provider job if it's still running and permanently deletes all of its checkpoints. This is irreversible — use `/stop` instead if you want to keep the checkpoints trained so far.
## Checkpoints and downloading weights
Pioneer saves checkpoints during training. You can list them at any point after the job starts:
```bash theme={null}
curl https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/checkpoints \
-H "X-API-Key: YOUR_API_KEY"
```
Each checkpoint carries `is_best`, `is_final`, and `is_deployable` flags. You can deploy any deployable checkpoint — not just the final one — to a live inference endpoint:
```bash theme={null}
curl -X POST https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/checkpoints/CHECKPOINT_ID/deploy \
-H "X-API-Key: YOUR_API_KEY"
```
To download weights instead, request a presigned URL (requires a Pro plan or above — this call returns `403` otherwise):
```bash theme={null}
curl https://api.pioneer.ai/felix/training-jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/download \
-H "X-API-Key: YOUR_API_KEY"
```
The response is JSON with a `download_url` that expires in 1 hour — fetch that URL separately to get the actual file:
```json theme={null}
{
"success": true,
"download_url": "https://...",
"expires_in_seconds": 3600,
"file_name": "my-ner-model-weights.zip"
}
```
You can also use a checkpoint UUID as the `base_model` value in a new training job to continue training from that checkpoint.
## Training endpoints summary
| Method | Endpoint | Description |
| -------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| `POST` | `/felix/training-jobs` | Start a new training job |
| `GET` | `/felix/training-jobs` | List training jobs (filter by `project_id`, `status`; paginate with `limit`/`offset`) |
| `GET` | `/felix/training-jobs/:id` | Get job status and metrics |
| `GET` | `/felix/training-jobs/:id/logs` | Get structured training log entries |
| `GET` | `/felix/training-jobs/:id/checkpoints` | List saved checkpoints |
| `POST` | `/felix/training-jobs/:id/checkpoints/:checkpoint_id/deploy` | Deploy a specific checkpoint for inference |
| `GET` | `/felix/training-jobs/:id/download` | Get a presigned URL to download trained weights (Pro plan+) |
| `POST` | `/felix/training-jobs/:id/stop` | Gracefully stop a running job, preserving checkpoints |
| `POST` | `/felix/training-jobs/:id/terminate` | Stop the job and permanently delete its checkpoints (irreversible) |
| `DELETE` | `/felix/training-jobs/:id` | Delete the job record — also stops it if active and deletes its checkpoints |