Signing tells you an artifact hasn’t been tampered with since it was signed. It tells you nothing about what went into creating it.

What signing actually gives you

OCI attestations solve a real problem. You can sign a ModelKit, attach scan results as attestations, and verify that the artifact in your registry is the same one that passed your security checks. Tools like Jozu Hub automate this: when you import a model or run a security scan, the scan result is a signed attestation attached directly to the artifact. No manual steps, no separate metadata store.

This is valuable. It’s also insufficient.

Single-artifact verification answers one question: is this artifact what it claims to be? It doesn’t answer: what went into producing it?

The fine-tuning problem

Consider three ModelKits in your registry:

• ModelKit A: Base model (registry.example.com/models/llama-base:v2), imported from HuggingFace. Jozu Hub has attached an import attestation recording its origin and a security scan attestation confirming the state of threat vectors.

• ModelKit D: Dataset (registry.example.com/datasets/customer-support:v1), curated internally. Jozu Hub attaches a security scan attestation including PII scores.

• ModelKit F: Fine-tuned model (registry.example.com/models/finetuned-support:v1), produced by fine-tuning A on D.

F has its own attestations. It was signed. It was scanned. But none of A’s provenance flows to F. None of D’s provenance flows to F. You can verify F in isolation and have no idea whether the base model was vetted or the dataset was scanned.

This is the gap.

Chains vs. graphs

Single-artifact provenance chaining is a known problem. Each attestation on an artifact can reference previous attestations, creating a linear history: built, then scanned, then approved. Tools exist for this. SLSA and in-toto define the formats. Rekor provides an append-only transparency log for ordering.

But fine-tuning isn’t a linear chain. F is produced from two inputs. Each input has its own attestation chain. The provenance structure is a graph, not a line.

F’s attestations need to explicitly capture its inputs — by digest — so that verification can walk the graph: confirm F, then confirm each input has its own valid chain. Without that link, the input attestations exist but are disconnected from the output. They’re metadata on separate artifacts that happen to be in the same registry.

Wait.. what about SBOMs?

If you’re thinking your SBOM already captures this relationship, you’re half right. An SPDX 3 SBOM on F can list A and D as inputs using relationship types like TRAINED_ON and GENERATED_FROM. It records versions and describes the dependency structure. That’s lineage — and it belongs in the SBOM. Jozu Hub can generate these. The composition is documented.

But the SBOM is a manifest, not a verification record. It describes what went into F. It doesn’t prove that the build process actually consumed those specific artifacts, and it doesn’t capture whether each input met its policy requirements at the time it was consumed. A signed SLSA provenance attestation does — it’s a cryptographic assertion from the build pipeline that says “I consumed these artifacts, at these digests, at this time.”

SBOMs own lineage. Attestations own proof. The SBOM tells a policy engine which inputs to check. The attestation on each input is what it verifies. But that verification — walking the graph and confirming each input met its requirements — is the piece that doesn’t exist yet.

Capturing input provenance at build time

This is the step most pipelines skip. When the fine-tuning job runs and F is packaged, the build process must record which artifacts were consumed, pinned by digest.

Fine-tuning is a build. The same SLSA provenance format works here. The buildType identifies it as fine-tuning, and resolvedDependencies captures every input artifact by digest.

Retrieve the digests before fine-tuning starts:

DIGEST_A=$(crane digest registry.example.com/models/llama-base:v2)
DIGEST_D=$(crane digest registry.example.com/datasets/customer-support:v1)

Then construct F’s provenance attestation:

{
  "buildDefinition": {
    "buildType": "https://jozu.dev/kitops/fine-tune/v1",
    "externalParameters": {
      "config": {
        "learning_rate": 2e-5,
        "epochs": 3,
        "batch_size": 16
      }
    },
    "resolvedDependencies": [
      {
        "uri": "registry.example.com/models/llama-base:v2",
        "digest": { "sha256": "abc123..." },
        "name": "base-model"
      },
      {
        "uri": "registry.example.com/datasets/customer-support:v1",
        "digest": { "sha256": "def456..." },
        "name": "dataset"
      }
    ]
  },
  "runDetails": {
    "builder": {
      "id": "https://jozu.dev/kitops/cli",
      "version": { "kit": "0.9.2" }
    },
    "metadata": {
      "invocationId": "ft-run-a7b8c9d0",
      "startedOn": "2025-03-15T09:00:00Z",
      "finishedOn": "2025-03-15T10:00:00Z"
    }
  }
}

Attach it to F:

cosign attest --key cosign.key \
  --predicate build-attestation.json \
  --type https://slsa.dev/provenance/v1 \
  registry.example.com/models/finetuned-support:v1

Now F’s attestation explicitly names its inputs using the same standard that import provenance already uses. No new predicate type. No custom schema. The graph exists. But verification is still on you.

Where cosign’s scope ends

Cosign verifies individual attestations. It confirms that a specific attestation is validly signed and attached to a specific artifact. It does not traverse the graph.

Graph verification requires additional logic:

1. Retrieve F’s provenance attestation and extract the resolved dependencies

2. For each dependency, resolve the digest in the registry and pull its attestations

3. Verify each attestation’s signature

4. Confirm the required attestation types are present — A needs both an import provenance and a scan attestation, D needs a scan attestation

5. Recurse if any input was itself produced from further inputs

No off-the-shelf tool does this today. The primitives are all there — cosign for signature verification, crane for digest resolution, OCI registries for attestation storage. The orchestration that ties them into recursive graph traversal is not.

Why almost nobody is doing this

Cosign doesn’t do it. The OCI spec doesn’t require it. Most pipelines treat each artifact in isolation — sign it, scan it, ship it.

The infrastructure to capture and verify input provenance has to be built deliberately. There’s no off-the-shelf tool that takes a fine-tuned model and walks back through its inputs to confirm each one was properly vetted. The primitives exist — digests, attestations, registries — but the orchestration layer doesn’t.

Part of the reason is that most supply chain security content stops at “sign your artifacts.” That’s Step 1. The harder problem — proving that the signed artifact was built from verified inputs — doesn’t have a standard solution yet.

What this prevents

Without input provenance capture, an attacker or a careless pipeline can substitute a different dataset or base model between approval and fine-tuning. F’s own attestations pass verification. The substitution is invisible.

With the graph in place, any substitution breaks verification. The digest recorded in F’s build attestation won’t match the artifact in the registry, or the artifact at that digest won’t have the required attestations. Either way, verification fails. That’s the point.

Signing is necessary. The graph is what makes it sufficient.

The provenance infrastructure described here is part of what we’re building into KitOps and Jozu Hub. ModelKits already carry the attestations. The next step is making input provenance capture and graph verification part of the pipeline, not an afterthought. If you’re working through similar problems, Yoi canfind me on LinkedIn.

Can the question be answered in 48 hours with cryptographic proof? Or does it require three weeks cross-referencing deployment logs, S3 timestamps, and spreadsheets—with gaps?

This is the provenance problem. Not just “where did this model come from” but “which data came from which model version, and can you prove it?”

The Real Problem: Two Disconnected Workflows

Traditional ML infrastructure treats model distribution and data collection as separate systems:

Models flow out: CI/CD pipelines, container registries, deployment scripts, version tags.

Data flows in: Custom collectors, S3 buckets, data lakes, ETL pipelines, manual tracking.

The connection between them? Spreadsheets. Naming conventions. Hope.

This breaks when you need to:

• Retrain models on data from specific versions

• Prove regulatory compliance with audit trails

• Debug performance issues across model versions

• Correlate field failures with deployment history

The root cause: models and data live in different systems with different identities. A model is medical-diagnostic:v2.1.4. Its training data is s3://bucket/device-042/2025-01-15/batch-003.parquet. There’s no cryptographic link.

The Solution: Bidirectional Artifact Flow

Treat both models and training data as versioned OCI artifacts with cryptographic references between them.

Outbound (Model Distribution): Models package as ModelKits—OCI artifacts containing weights, inference code, and configuration. Edge devices pull by content-addressable digest: sha256:abc123...

Inbound (Data Collection): Training data packages as separate ModelKits that reference the source model by digest. These Training Data ModelKits contain inference outputs, optional ground truth labels, and metadata (device ID, timestamps, inference parameters).

Single Infrastructure: The same OCI registry handles both directions. Models distribute outbound. Training data flows inbound. Query by model digest to aggregate data from specific versions across your entire device fleet.

Cryptographic Chain: Sign and attest both artifact types. The provenance link is mathematically verifiable, not administratively maintained.

How It Works

Model Deployment

Inference and Data Collection

Retraining Query and Aggregation

The beauty: models and data share the same identity system. Query by model digest returns all training data generated by that version across your entire fleet. The provenance is in the artifact structure itself.

This Works Today

The pattern relies on standard OCI infrastructure and existing tools. KitOps handles ModelKit creation and streaming pack-and-push. Any OCI-compliant registry works, though registries with metadata query capabilities (like Jozu Hub) simplify data aggregation. Cosign or Notary provide signing.

The architecture scales with your registry infrastructure. If you’re deploying models to edge devices in regulated environments—automotive, medical devices, industrial control systems—the provenance problem isn’t optional. Auditors will ask which model version produced which results. This architecture answers that question with cryptographic proof instead of spreadsheets.

The bidirectional flow isn’t novel infrastructure. It’s the logical extension of treating ML artifacts the same way we learned to treat application artifacts: versioned, immutable, content-addressable, and cryptographically verifiable.

Subscribe now

Momentum is growing

Let me walk through what’s actually happened over the last 18 months, because the timeline tells a story about ecosystem convergence:

The momentum started quietly. When Jozu released KitOps in March 2024, early users responded: “Why didn’t we think of this sooner?” By May 2025 it entered the CNCF Sandbox and has seen over 150K downloads. KitOps offers a Docker-like CLI ( kit pack, kit push, kit pull) plus native Hugging Face integration, letting teams import models into OCI registries with automatic metadata generation.

In June 2025 the CNCF accepted ModelPack into its Sandbox, marking the first vendor-neutral open standard for packaging ML artifacts as OCI objects. Contributors include Jozu, Red Hat, PayPal, ANT Group, and ByteDance—indicating commitment beyond startups. ModelPack builds directly on the OCI Image Manifest Specification v1.1, defining custom media types for models, datasets, code, and documentation as separate deduplicated layers.

Docker shipped Model Runner in beta (April 2025) and reached general availability in September 2025. It introduces an OCI artifact approach for GGUF-format LLMs, treating models as first-class artifacts with domain-specific configuration schemas.

Red Hat’s OMLMD project (July 2024) delivers a Python SDK and CLI for working with ML models and metadata via OCI registries.

Subscribe now

Kubernetes integration establishes the foundation

Kubernetes 1.31 (August 2024) introduced Image Volume Source (KEP-4639) in alpha—native support for mounting OCI artifacts as read-only volumes in pods, specifically designed for AI/ML use cases. This eliminates the init-container pattern previously required to fetch models, instead allowing data scientists to package model weights as OCI objects that mount directly alongside model servers. KServe added OCI artifact support and KitOps released its integration with KServe with support of multiple cloud providers.

OCI specifications for AI/ML artifacts have achieved genuine production readiness as of October 2025. Technical implementation is mature with OCI v1.1 providing robust architectural foundations. Security tooling spans ML-specific scanning, signing, attestations, and policy enforcement.

Two OCI specifications have emerged serving distinct contexts:

• Docker Model Runner: targets desktop / local inference workflows; emphasizes GGUF-format models optimized for consumer hardware ergonomics.

• CNCF ModelPack: targets enterprise production; supports multiple model formats (SafeTensors, ONNX, PyTorch) plus governance artifacts (attestations, SBOMs, policy bundles).

Should you adopt this now?

OCI artifacts for ML make operational sense today if you are:

• Running Kubernetes-native infrastructure

• Under regulatory or internal compliance pressure for model provenance

• Distributing models across multiple teams or external partners

• Already container-centric in your release and operations workflows

If none of these apply, keep your current workflows—S3 buckets and DVC remain perfectly valid.

If you’re evaluating adoption and want to compare implementation patterns—what’s working and where things still break—send a message below (or comment). If you’re building tooling in this space, I’d love to hear what you’re seeing.

✅ The Good

When a monorepo works, it works because the alignment and velocity benefits outweigh any drawbacks. Here is what I noticed:

• Shared context: Everyone, from data scientists to platform engineers, had visibility into the same repository, which fostered fast collaboration and fewer misunderstandings.

• Fast iteration: A data scientist could tweak a model and then message an engineer to wire it up to an API within the same codebase.

• Unified CI/CD: Teams could run pipelines for end‑to‑end tests, integrate model‑training jobs into GitHub Actions, and deploy inference microservices using the same scripts.

❌ The Bad

This setup had major flaws, some of them critical to production readiness.

• No model provenance: Models trained in notebooks were often dumped into S3 buckets with ad‑hoc names. There was no versioning or traceability. The teams included their names in the model filenames, but that practice did not age well.

• Reproducibility gaps: Because experiments were often driven from notebooks, they lacked pinned dependencies or runtime configuration. Rerunning a past experiment was, at best, guesswork.

• Security blind spots: With no SBOMs or attestations, the security team had no idea what was running in production, creating a compliance risk.

😬 The Ugly

Some things technically “worked,” but only through tribal knowledge, individual heroics, or duct‑taped workflows.

• Manual model handoffs: Data scientists pinged infrastructure engineers on Slack with pointers to model files. There was no formalized way to package a model.

• Inconsistent naming conventions: Some model folders were named teamXXX_final_modelv2, while others used names like modelXX_2024_05_19. Pipelines frequently broke when a model name changed or a new model appeared.

• Overloaded CI pipelines: A single Git push could retrigger training, redeploy the inference container, and run unrelated tests. The infrastructure was brittle because the monorepo lacked boundaries between experimentation and production.

• Blurred ownership: When a model in production failed, nobody knew whether to call the data scientist, ML engineer, or platform SRE. The repository did not encode accountability.

🧰 How KitOps Helped

KitOps introduced structure at the artifact level without forcing the team to refactor the entire repository.

1. Clear handoff via ModelKit artifacts
   Data scientists used the kit CLI and pykitops to export trained models as self‑contained, versioned ModelKits that included:

   • Weights

   • Metadata (input and output schema)

   • Optional model cards as README files

   • Runtime dependencies (for example, tokenizers, configuration files, and sometimes Python code)

   These kits became immutable units that downstream teams could trust.

2. Decoupled training and inference
   ModelKits were pushed to an OCI‑compatible registry where inference microservices could pull them at runtime. Training scripts no longer needed to be bundled into deployment images. The same model could be pulled into staging, production, or offline evaluation environments with confidence, allowing platform engineers to treat inference containers as cattle rather than pets.

3. Auditability and compliance
   The team did not yet add SBOMs to ModelKits, but they recorded the monorepo SHA as an attestation with which each ModelKit created. This practice gave the security team visibility into what was running in production and where it came from, easing a key compliance bottleneck.

4. Standardization without a repo rewrite
   The team adopted one simple convention: if a model is going to production, it must be exported as a ModelKit. That rule turned Git chaos into structured deployment boundaries.

Monorepos are a double‑edged sword. Their collaboration benefits are impressive, but scaling them, especially in AI and ML systems, requires discipline.

KitOps did not “fix” the monorepo; arguably, it did not need fixing. Instead, it created clean seams where they mattered most: at the handoff between teams and in the lifecycle from experimentation to production. That was enough.

At Jozu, though, we’re solving the same production inference problem—but from the perspective of platform and DevOps teams, not just model consumers.

🎯 The Problem We’re Solving: Infra Teams Need Control, Not Just Speed

NIMs are fantastic for what they do: help teams quickly run high-performance inference workloads on NVIDIA hardware. But many of the teams we work with—infra, platform, and SRE leads inside organizations scaling their AI infrastructure—need something more composable, secure, and lifecycle-aware. They need:

• Full control over which model goes where and how it’s updated

• Clear provenance and auditability across environments

• The ability to decouple containers from models, just apps were decoupled from hardware

That’s why we built Jozu Rapid Inference Containers (RICs) —a packaging and runtime architecture that treats models as first-class, injectable artifacts, not frozen blobs inside prebuilt containers.

🧱 What Are Jozu RICs?

At a high level, RICs are base inference containers (e.g., using vLLM, Triton, llama.cpp, etc.) that dynamically inject the right model at pull-time using our open packaging format called ModelKits.

A ModelKit includes:

• Model weights (e.g., GGUF, safetensors, ONNX)

• Optional adapters (e.g., LoRA)

• Inference config (Triton, vLLM, etc.)

• Metadata, licenses, SBOMs, and attestations

🧠 Models ≠ Containers

Most model serving setups today — including NVIDIA NIMs — bundle the model directly into the container image. That might feel convenient, but it tightly couples what you serve (the model) with how you serve it (the container).

This creates real problems:

• Promoting a model to production means building and shipping a new container.

• Rolling back a model requires knowing which image it was embedded in.

• You can’t track or attest to models independently of the containers they're wrapped in.

Jozu RICs break this coupling.

We treat the container as the runtime, and the model as a pluggable artifact, delivered just in time at pull.

This gives you major advantages:

• ✅ Run the same container across staging, production, or air-gapped environments — just inject different models.

• ✅ Promote, attest, or roll back individual models without touching containers or deployments.

• ✅ Automate model lifecycle using GitOps: model changes are observable, auditable, and versioned like application code.

Think of it like Helm charts for models—one runtime, many artifacts, full control.

⚖️ Where NIMs Fall Short—and What Jozu Solves Instead

To be fair — if you’re a product team at an enterprise shipping a known, stable model behind an API, NIMs might just work.

They’re great for:

• Use cases that only run on NVIDIA GPUs

• Scenarios with a few approved models reused broadly

• Teams that prioritize convenience over lifecycle governance, customization, or traceability

We’re not here to replace NIMs for those teams.

But if you're asking:

• How do I track where every model came from?

• How do I promote a fine-tuned adapter without repacking a container?

• How do I plug models into Kubernetes-native CI/CD?

Then you’re thinking the same way we are.

🔒 The Bigger Picture: Secure ModelOps

Model security isn’t just about scanning containers anymore. It’s about:

• Provenance: Where did this model come from? Who trained it?

• Reproducibility: Can I build it again, exactly?

• Attestation: Can I prove no one tampered with it?

That’s what RICs and ModelKits are built for — and where we think the industry is headed.

✅ Want to Try It?

Jozu’s CLI (kit) is open source. You can:

• Package your own models as ModelKits

• Use RIC-compatible base images

• Deploy via Kubernetes and GitOps

We’d love feedback — especially if you’ve used NIMs and hit their limits.

NVIDIA NIMs and Jozu RICs are both trying to solve a hard problem: how to bring models to production without chaos.

If you're looking for a simplified start with NVIDIA’s ecosystem, NIMs offer a convenient starting point — but with trade-offs in flexibility and control.

If you want lifecycle-aware, security-conscious, DevOps-aligned model delivery, Jozu RICs might be the alternative you didn’t know you needed.

This isn’t a knock on Docker. It’s a recognition of its scope. If your goal is to run a model on your laptop or share it with a teammate, Model Runner might suffice. But if you're serious about advancing models through development, staging, and production pipelines, you need something much more robust.

Where Docker Model Runner Falls Short in AI Workflows

Docker containers are excellent for application-level reproducibility. But AI models are not applications—they’re evolving systems that require:

• Datasets, training configurations, and preprocessing logic

• Dependencies tied to specific frameworks and hardware

• Lifecycle management: versioning, governance, model evaluation, rollback

• Operational needs: GPU scheduling, autoscaling, routing, monitoring

Docker’s Model Runner abstracts only the model weights, enough to get started, but insufficient for production use.

ModelKits: Purpose-Built for AI

KitOps introduces ModelKits: immutable, composable packages that include not just model weights, but everything needed to reproduce and operate that model across environments.

Each ModelKit includes:

• Model binaries

• Training and inference code

• Configuration files

• Datasets

• Evaluation results

This is more than packaging—it’s reproducibility and lifecycle control, baked into the artifact.

What ModelKits Add to the Picture

Docker’s Model Runner is a helpful starting point—a general-purpose tool now being applied to the evolving domain of AI models. It’s designed for:

• Local usage

• Simplified packaging

• Ad-hoc inference serving

ModelKits are designed for:

• Full ML lifecycle reproducibility

• Multi-stage workflows and pipelines

• Collaboration across research, infra, and product teams

• Seamless transitions across dev, staging, and production

• KitOps supports local-first workflows too. With kit dev, you can spin up an inference server

When paired with a registry and KitOps tooling, ModelKits integrate natively with CI/CD pipelines, distributed inference runtimes, and governance platforms.

Shipping AI Products? Choose the Right Foundation

If you're solo, prototyping, and just need a quick way to serve a model—Docker’s Model Runner will work.

But if you’re running real experiments, versioning datasets, collaborating with others, deploying across environments, or managing audit trails—you need something built for the job.

You need KitOps.

You need ModelKits.

Because packaging the model is just the beginning.

What happens after is where real AI engineering begins.

Let’s unpack that.

🛠️ Git Is Great for Development

Tools like DVC, Hugging Face Hub, and Git LFS are excellent for development workflows. They’re optimized for:

• Rapid experimentation

• Versioning models alongside code

• Collaborative iteration between data scientists

These tools assume a development mindset: mutable history, active collaboration, and quick rollback. That’s perfect when you're training, tuning, and testing—but it starts to fall apart when you're preparing for production.

🚧 Git Breaks Down at Deployment

Once you're ready to serve models in production, your requirements change:

• You need immutability, not editable branches

• You need to promote the exact same artifact across dev, staging, and prod

• You care about provenance, signing, and attestations

• Your CI/CD system must treat models and datasets as first-class deployable units

Git wasn’t built for this. Teams often end up with brittle workflows: custom scripts, out-of-band state management, and inconsistencies that are hard to debug.

✅ OCI Artifacts Are Built for Deployment

Enter OCI artifacts—the same standard used to ship Docker containers. But OCI is broader than that. It’s a general-purpose packaging format that supports any binary blob, including model checkpoints, datasets, and structured metadata.

KitOps leverages OCI to package ModelKits: versioned, self-contained AI/ML artifacts with metadata, optional attestations, and digital signatures. This brings deployment maturity to model management:

• Immutable by digest

• Promotable via tag and digest separation

• Verifiable using Cosign, Notation, or Sigstore

• Integrates with policy engines like Kyverno and Gatekeeper

These aren’t theoretical advantages. OCI artifacts are hardened, production-grade units. With KitOps, those same principles apply to models and data pipelines.

🧩 Where Git Ends and KitOps Begins

Here’s how responsibilities divide across the ML lifecycle:

Some teams attempt to stretch Git + DVC all the way to production, but that often involves fragile CI logic, lockfile gymnastics, and deep institutional knowledge. OCI-native workflows avoid these traps by building on standard, cloud-native patterns.

This isn’t just a packaging debate—it’s about ecosystem fit. OCI artifacts integrate seamlessly into Kubernetes-centric environments. They can be signed, scanned, validated, and promoted using the same pipelines and tooling already adopted by DevOps and platform teams.

🧠 Final Thought

If you're deploying models by cloning Git repositories and running bash scripts, you're using dev tools to solve a deployment problem. That might work—until it doesn't.

Infrastructure teams already use OCI to ship software. KitOps brings that same discipline to AI/ML. With OCI-based ModelKits, you're not just deploying models—you're treating them like the production software they are.

The Genesis: Attestations, Provenance, and Immutability

What Are Attestations?

An attestation is a verifiable piece of metadata that certifies specific properties or events related to an AI/ML artifact. This can include information about how the model was built, which datasets were used, how the dataset was collected, or simply the source.

Attestations help establish trust by providing a record of the processes and checks that the AI/ML artifacts have undergone. They offer multiple benefits:

• Enhanced Security: Attestations help ensure that artifacts haven’t been tampered with and that they originate from a trusted source.

• Auditability: They provide a clear, verifiable record of an artifact’s history, which is valuable for auditing and forensic investigations.

• Compliance: In regulated industries, attestations can serve as evidence that an artifact meets required standards and policies.

• Operational Confidence: By verifying attestations before deployment, organizations can have greater confidence that only approved and vetted artifacts are running in production.

What Is Provenance?

Provenance refers to the comprehensive record of an artifact’s origin, history, and the processes it has undergone from its creation to its current state. In the context of AI/ML artifacts (or other software artifacts), provenance may include details such as the origin of the model (for example, a hash or URL), the training process including the datasets used, lifecycle events such as validations and security checks, and the chain of custody.

The Relationship of Attestations and Provenance

Attestations and provenance are closely related, yet they serve distinct purposes within a secure supply chain. Provenance acts as a detailed record of the artifact’s lifecycle, capturing every significant step, transformation, or interaction that the artifact experiences. It provides a complete historical record, while attestations serve as formal, verifiable assertions about the artifact’s state or properties at a specific point in time. An attestation might assert that the artifact meets certain criteria or has passed specific tests.

Common Secure Supply Chains Scenarios

• Rapid Verification: In many automated environments, verifying a signed attestation is quicker than reconstructing the entire provenance.

• In-Depth Auditing: If a potential security issue arises, the provenance data can be reviewed in depth to understand the artifact’s full history.

• Regulatory Compliance: Some standards require both a summary assertion (attestation) and a detailed audit trail (provenance) to meet strict compliance requirements.

What Is Immutability?

Immutability refers to the property of an artifact or data structure that prevents it from being changed after its creation. Immutability plays a crucial role in the effectiveness and trustworthiness of attestations and provenance.

1. Consistency Between Attestation and Artifact

   When an artifact is immutable, it guarantees that the conditions and properties described in the attestation (such as build parameters, security scans, or provenance details) remain accurate over time. Since the content is fixed, a verifier can confidently check that a digest or hash matches the one recorded in the attestation. Any deviation indicates tampering or an inconsistency, leading to a clear rejection of the artifact.

2. Enhanced Security Posture

   In a secure supply chain, immutability combined with a signed attestation creates a robust chain-of-custody. Each artifact remains exactly as it was when it passed through each stage of the pipeline, reducing the risk of undetected modifications.

3. Simplified Compliance and Audit Processes

   Immutable artifacts, paired with their corresponding attestations, provide a consistent and verifiable audit trail. Auditors can rely on the fact that the artifact's properties haven’t changed since the attestation was issued. Many regulatory frameworks require strict traceability and non-repudiation of changes.

4. Operational Stability

   Immutability ensures that only approved and vetted artifacts are running in production. If an issue arises, the assurance that the artifact is immutable allows operators to roll back to a previous, known-good state without worrying about hidden modifications.

How It All Works for ModelKits

Generation

In an AI/ML pipeline—whether during training, fine-tuning, or data extraction— the first step for secure packaging is creating a ModelKit. As the ModelKit is generated, an accompanying attestation can be produced that captures essential metadata.

The attestation metadata might include details such as the build environment, the results of scans, or the outcome of compliance tests. This attestation is often signed using cryptographic keys, ensuring that it comes from a trusted source and has not been tampered with.

How to Create Attestations for ModelKits

1. Package Artifacts as a ModelKit and Automate Its Generation.

   Package models, datasets, code, and documentation to create an immutable package and store it on a secure OCI registry.

   • Tools: Use the kit CLI or pykitops to automate creation of ModelKits on your pipelines.

   • Automate: Include this step in your AI/ML pipelines so that every artifact that is a candidate for production is a ModelKit.

2. Define Essential Metadata

   Determine which details are critical for your attestation. Common metadata elements include:

   • Build Environment: Information about the operating system, tools, and configuration used during the build.

   • Security and Compliance Checks: Results from vulnerability scans, static analysis, or other compliance tests.

   • Pipeline Steps: A record of the build and test stages executed.

   • Timestamps and Identifiers: The build date/time and unique identifiers (e.g., a hash or version tag) for the ModelKit.

3. Automate Attestation Generation

   Incorporate attestation creation into your pipeline by automating the process:

   • Scripting and Tools: Use scripts or dedicated tools to automatically collect and format the metadata. (for example, output the data in JSON or YAML).

   • Integration: Embed this step within your existing CI/CD workflow (using systems like Jenkins, GitLab CI, or GitHub Actions) so that every build triggers an attestation generation.

4. Sign the Attestation

   To ensure that the attestation is both authentic and tamper-evident:

   • Digital Signing: Use cryptographic keys to sign the attestation. This confirms that it was produced by a trusted source.

   • Key Management: Securely manage your private keys—tools like Cosign can help streamline signing and verification processes.

Storage and Association

The generated attestation is then associated with the ModelKit. This can be done by storing it in an OCI registry alongside the ModelKit or in an external attestation store. Tools and standards such as Cosign or Notary can be used to facilitate this association.

Verification

Before deploying a ModelKit, systems (like a Kubernetes admission controller or a CI/CD pipeline) can verify the attestation. Verification ensures that the ModelKit was built under the expected conditions, and that the metadata hasn’t been altered. It checks the digital signature against a trusted key. If the attestation is valid, it can be trusted to meet the security and compliance requirements. If not, the deployment can be halted or flagged for further review.

How to Create, Sign, Store, and Verify a Simple Attestation Using Cosign

These instructions assume that you have just created a ModelKit, tagged it with jozu.ml/myorg/mymodel:latest, and pushed it to the OCI registry.

Generate a Key Pair with Cosign
Cosign makes it easy to generate a key pair. This command creates two files: a private key (cosign.key) and a public key (cosign.pub).

cosign generate-key-pair
# Consider storing keys in a secure location or HSM and set up key rotation policies.

Generate Attestation
Create an attestation file containing metadata about your ModelKit. For example:

ARTIFACT_ID="mymodel-123abc"
BUILD_ENV=$(uname -a)
TRAINING_PARAMS="learning_rate=0.01, epochs=50"
TIMESTAMP=$(date)

cat <<EOF > attestation.json
{
    "artifact_id": "$ARTIFACT_ID",
    "build_environment": "$BUILD_ENV",
    "training_parameters": "$TRAINING_PARAMS",
    "timestamp": "$TIMESTAMP"
}
EOF

This script generates an attestation.json file containing some example metadata. It is recommended to use a well known attestation format in real-world cases.

Sign the Attestation and Attach It to Your ModelKit

cosign attest --key cosign.key --predicate attestation.json --registry-username=<registry_user> --registry-password=<registry_pass> jozu.ml/myorg/mymodel:latest

Verify the Attestation
To confirm that your attestation has been correctly signed and stored, use the verification command

cosign verify-attestation --key cosign.pub jozu.ml/myorg/mymodel:latest

This command retrieves the attestation from the OCI registry, verifies its digital signature using the public key, and displays the attestation details if the verification is successful

Standards and Formats for Attestations

A critical element of a secure supply chain (SSC) is the standardization of how attestations are formatted, signed, and verified. By adhering to industry standards, organizations ensure that the metadata accompanying artifacts is both interoperable and verifiable across different systems and tools. Below are some of the most prominent standards and formats used in the SSC landscape:

• in‑toto: in‑toto is a framework designed to capture the complete supply chain of an artifact. It records every significant step—from code commit to artifact generation—in a detailed, cryptographically verifiable manner. The in‑toto format allows for complex workflows to be attested, ensuring that each stage of the build, test, and deployment process is recorded in an immutable log. This depth of detail is invaluable in environments where understanding the full history of an artifact is crucial for trust and compliance.

• DSSE (Dead Simple Signing Envelope): DSSE provides a lightweight, standardized container for signing arbitrary JSON payloads. It is designed to encapsulate attestation data in a manner that is both simple and secure, making it easier to integrate into automated pipelines. DSSE’s simplicity and focus on digital signature management help maintain the integrity of the attestation, ensuring that any tampering can be quickly detected. Tools like Cosign use DSSE to wrap attestations, providing a consistent method for signing and verifying artifact metadata.

• OCI Image Attestation Specification: In environments where artifacts are stored in OCI-compliant registries (e.g., container images), the OCI image attestation specification plays a vital role. This standard allows attestations to be stored alongside artifacts in the registry, linking metadata directly to the artifact via its digest. The format is tailored for containerized environments, ensuring that the provenance, security, and compliance of container images are verifiable before deployment.

Our example has not leveraged the in-toto framework but rather used a simple JSON document to better demonstrate the process. However, by leveraging these standards, organizations can create a robust framework for ensuring that every ModelKit or AI artifact is accompanied by trustworthy attestations. These formats not only facilitate secure signing and verification processes but also enhance interoperability between different tools and platforms, thereby strengthening the overall security and auditability of the software supply chain.

Alignment with Regulatory Standards

Having explored the core technical mechanisms behind ModelKits—from generating secure artifacts and signing attestations to ensuring immutable storage—we now turn our focus to a critical aspect of secure supply chains: regulatory compliance. In today’s complex security landscape, aligning with frameworks like NIST, ISO, and GDPR is not only beneficial but essential. The following section delves into how ModelKits meet these standards, providing concrete examples and a comparative overview of key compliance features.

Ensuring regulatory compliance is critical for organizations leveraging ModelKits in their secure supply chain. By aligning with established frameworks such as NIST, ISO, and industry-specific standards, ModelKits not only help protect critical assets but also streamline audit processes and support forensic investigations.

• NIST SP 800-161 (Supply Chain Risk Management):
  ModelKits incorporate cryptographic signatures and immutable logs to ensure traceability and non-repudiation. These features support NIST requirements for tracking and verifying the origin and integrity of artifacts. The comprehensive provenance data maintained throughout the pipeline also aids in risk assessment and incident response.

• ISO/IEC 27001 (Information Security Management):
  With built-in mechanisms for secure artifact storage, digital signing, and continuous verification, ModelKits support ISO/IEC 27001 objectives by ensuring that only approved, tamper-evident artifacts are deployed. The structured metadata and audit trails facilitate regular security assessments and help demonstrate compliance during certification audits.

• GDPR (General Data Protection Regulation):
  Although GDPR primarily focuses on data privacy, ModelKits can play a role in ensuring that data handling processes are auditable and that any data transformations or access events in the AI/ML pipelines are properly logged. This helps organizations enforce data protection policies and manage consent-related requirements.

• Industry-Specific Regulations:
  For sectors such as finance, healthcare, or critical infrastructure, ModelKits can be tailored to capture compliance-specific metadata (e.g., financial audit trails or patient data access logs), thus meeting stricter regulatory requirements.

By integrating compliance considerations into every stage—from artifact generation to verification—ModelKits provide a solution for organizations looking to meet diverse regulatory requirements. The use of standardized attestations and immutable storage not only fortifies the security of the software supply chain but also offers clear, verifiable audit trails that are essential for compliance with NIST, ISO/IEC 27001, GDPR, and other regulatory frameworks.

Why MCP Matters

Given my experience with LSP, I’m enthusiastic about the growing interest in the Model Context Protocol (MCP). However, I am concerned that the valuable lessons learned from LSP are not being effectively applied to MCP.

When LSP emerged, it transformed programming language tooling. Specifically, it allowed language experts to implement sophisticated, language-specific intelligence consistently across different IDEs and editors. LSP created an abstraction enabling the same compiler development teams to directly support any IDE or editor.

MCP provides an analogous abstraction between AI tools and agents and their computing environments. However, the type of abstraction provided by LSP—deep, specialized programming language expertise—is significantly more complex to integrate and replicate compared to the API interactions primarily targeted by MCP. This difference currently makes MCP’s value proposition lower than that of LSP, which raises ongoing questions about whether MCP provides substantial value beyond existing APIs.

Critical Risks with Current MCP Implementations

Unfortunately, MCP carries forward several critical shortcomings that were also issues with LSP. One significant oversight with LSP was the lack of standardized packaging. Visual Studio Code—the hero product driving LSP adoption—provided its own method for packaging extensions, but this approach was not easily transferable to other platforms. The absence of standardized, secure packaging made LSP implementations vulnerable to supply chain attacks. Even VS Code’s extension packaging was not originally designed with supply chain security in mind, proving vulnerable at times.

The risk is even greater with MCP due to its broader potential access and integration to critical systems. Organizations face significant security risks if they adopt MCP directly from third-party sources without a robust packaging solution that includes secure attestations and digital signatures.

Additionally, LSP is defined to operate on single-user desktop environments without built-in multi-tenancy, a feature that simplifies implementation, but limits use in cloud environments. This lack of multi-tenancy poses a much larger challenge for MCP, as MCP implementations are more likely to run in multi-tenant environments requiring robust authentication and authorization.

Without addressing these critical issues related to packaging, secure supply chains, multi-tenancy, authentication, and authorization, the overall value and viability of MCP will continue to be questioned.

At Jozu, we are uniquely positioned to address these critical MCP adoption challenges. With extensive experience gained from pioneering work on LSP and our development of KitOps—a proven open-source solution trusted by enterprises for securely packaging and deploying AI/ML workloads—we are prepared to solve MCP’s most pressing security and packaging issues. Partnering with us will help your organization significantly reduce exposure to supply chain risks while accelerating secure MCP adoption.

Your Opportunity: Become a Design Partner

We’re currently seeking a limited number of design partners to join us in shaping the future of MCP. As a design partner, you’ll gain exclusive access to our solution, have direct influence on product direction, and receive expert guidance on securely implementing MCP in your organization.

Spots are limited—contact today to secure your position.

Environmental changes have always been catalysts for evolutionary shifts. The rise of Large Language Models (LLMs) like ChatGPT has ignited the birth of a new technological ecosystem. But, as with any seismic change, the initial response is often wildly exaggerated, driven by those who don't fully grasp the nuances—a phenomenon we now dub the *AI Hype*. For those of us who've seen such waves before, it was clear from the start: this was a hype cycle, and like all hype cycles, it had to run its course. Now, the signs are undeniable—the hype is cooling down. But what's next for AI?

Subscribe now

No technology, no matter how revolutionary, can thrive without delivering real value. And here's the truth: we haven’t yet cracked the code on generating consistent value for enterprises from AI. The next phase of AI's evolution won't be about shiny new algorithms or eye-catching demos; it will be about the nitty-gritty work of building standards, developing techniques, and creating frameworks that enable AI and ML to integrate into the fabric of business seamlessly and safely. We've begun to see pockets of success and early experiments that hint at the financial benefits AI/ML can offer. But as the hype bubble bursts, there's always a danger that the technology itself will be blamed for the inevitable disappointments—rather than the non-specialists who inflated expectations to begin with.

The reality is clear: AI/ML works. It has proven, valuable applications and should not be discarded just because it was overhyped. Now, more than ever, it’s time to rally behind standards, support open-source initiatives, and back companies that are focused on streamlining the adoption of AI and ML. This is how we accelerate the next phase of AI's evolution, turning what was once hype into sustainable, real-world impact.

Leave a comment

What is a Control Plane?

Before we delve into the specifics of a Control Plane for AI/ML, let's define what a control plane is. A control plane is responsible for managing and orchestrating various components within a system. It is distinguished from the data plane, which handles the actual data processing and forwarding tasks. The control plane functions as the “brain” of the system, providing instructions, managing configurations, and ensuring that the various components operate cohesively. Examples of control planes include the Kubernetes Control Plane, which manages the lifecycle of containers across a cluster, or in Software-Defined Networking (SDN) where SDN controllers like OpenDaylight, ONOS, and Cisco APIC provide a centralized view of the network.

A Control Plane for AI/ML

Machine learning workflows are complex, multi-staged, nondeterministic, and continuous, making them prime candidates for huge benefits from control planes. A control plane that orchestrates various stages of the AI/ML lifecycle, including data preprocessing, model training, deployment, monitoring, and maintenance, vastly improves day 1 and day 2 operations for AI/ML applications.

Key Components of an AI/ML Control Plane

1. Data Pipeline Management: Ensures smooth data flow from ingestion to processing and storage, managing dependencies and scheduling tasks.

2. Model Training and Deployment: Coordinates the training of models, manages model repositories, and handles deployment to various environments (e.g., cloud, edge).

3. Resource Allocation and Optimization: Dynamically assigns computational resources based on workload demands and priorities, optimizing for cost-efficiency and performance.

4. Monitoring and Logging: Provides real-time insights into the performance and health of models and infrastructure, enabling proactive issue resolution and continuous improvement.

5. Security and Compliance: Enforces security policies, manages user access, and ensures compliance with data privacy laws and industry standards.

Each of these components can encompass multiple concerns. For instance, resource allocation and optimization can deal with both scaling and cost management, while security and compliance can cover access management and auditing. Furthermore, there can be multiple tools, like MLflow, feature stores, and monitoring solutions present on the data plane for each of these areas that the control plane would have to work with. Therefore, a superior quality for a control plane is its extensibility to accommodate different concerns and tools.

Why Do You Need One Now?

If you do not already have an application control plane, chances are you have been using some ad-hoc scripting with manual monitoring in addition to good DevOps practices such as Infrastructure as Code and CI/CD pipelines. Depending on the size and complexity of your operation, you may already feel the need for a control plane and even be building a custom one slowly. Unfortunately, AI/ML workloads do not bring good news. AI/ML applications come with the following new or magnified issues:

• Increased Complexity: Managing AI/ML workflows is more complex and fragmented, often involving multiple stages leading to potential inconsistencies and errors.

• Scalability Issues: Scaling operations to handle larger datasets, more complex model workloads, and high computational demands is challenging.

• Resource Inefficiency: Resource underutilization is more costly due to the need for more expensive resources.

• Monitoring and Logging: New requirements like detecting model and data drift, interoperability, and explainability of AI models, and increased security and compliance requirements arise.

• Security and Compliance Risks: AI and ML applications bring new security and compliance risks such as ethical considerations, transparency and explainability, data privacy and security, and model security.

All these issues are interrelated. Without an extensible platform to manage the complexity, trying to solve these issues will feel like plugging one leak while another pops up.

The Time to Act is Now

For most enterprises, competitive edge in leveraging AI/ML technologies effectively hinges on the ability to manage and optimize these processes seamlessly. Taking immediate steps to address these challenges will prevent your AI/ML initiatives from becoming mired in inefficiencies and compliance risks. Ensure your enterprise remains competitive by implementing a robust AI/ML control plane now. My team at Jozu is working on such an extensible platform that can help you and we would like to hear from you.

Enterprises should not be on the hunt for solutions that are solely AI-based. Instead, they should be seeking solutions that are AI-infused. There’s a subtle but significant difference. Generative AI and advanced machine learning techniques are remarkable, yes, but they are not panaceas. These technologies should be viewed as powerful tools to enhance and integrate within existing systems, rather than standalone solutions that promise instant transformation.

Infusing AI into business processes can indeed offer substantial benefits—enhanced efficiency, better decision-making, and improved customer experiences, to name a few. However, the real challenge lies in the integration. It’s not just about having the latest AI models; it’s about embedding them into the fabric of enterprise systems in a way that drives tangible business outcomes.

This is where the industry faces a bottleneck. There’s a scarcity of professionals who possess a trifecta of expertise: a strong command of AI and machine learning, adept application development skills, and an understanding of business dynamics. Designing AI-infused systems requires a multidisciplinary approach. It’s not enough to be a master of one domain; the true innovators in business will be those who can bridge the gaps between these critical areas.

It might sound paradoxical coming from the CTO of an AI/ML company, but this is precisely why we established our venture Jozu. Our mission is to demystify and democratize AI/ML for enterprises and application developers. We aim to make it easier for businesses to harness these technologies, not as isolated wonders, but as integral components of their operational infrastructure.

The journey toward meaningful AI integration is nuanced and complex. But by focusing on AI infusion rather than chasing after standalone AI solutions, enterprises can position themselves to unlock the real value these technologies offer. It’s a path of incremental enhancements, strategic implementations, integration with current operations and continuous learning—an approach that promises a more sustainable and impactful transformation.

Share

Technical and Economic Barriers to Adoption

Foremost among the barriers to AI integration is the formidable cost of entry. The infrastructure required to train and deploy sophisticated AI models is not trivial; it encompasses advanced computational resources like high-throughput GPUs and extensive data sets, necessitating significant capital investment. Beyond the hardware, the ecosystem's lack of homogenized frameworks and standards magnifies the operational costs and complicates lifecycle management of AI assets.

The Imperative for Standardization

In the absence of universally accepted standards, fundamental decisions—ranging from model packaging to dataset structuring—become potential future liabilities. The rapid obsolescence rate within AI technology domains means today’s cutting-edge is a legacy system next month. Selecting an inference engine or committing to a specific runtime environment often locks developers into rigid architectures that resist modular updates or cross-platform interoperability.

Moreover, the proliferation of proprietary platforms by dominant market players fosters a ‘walled garden’ environment. This not only stifles innovation by curtailing cross-vendor operability but also risks significant strategic assets being perennially tethered to a singular vendor’s ecosystem.

Strategic Initiatives and Open Standards

This landscape, however, is not sculpted by deliberate exclusionary practices but rather by a nascent field evolving through its teething phases. Visionary technologists and industry pioneers are acutely aware of the benefits of standardized AI practices—enhanced adoption, reduced entry barriers, and greater innovation.

Initiatives such as ONNX, KitOps (which I spearheaded), embody this ethos by advocating for and implementing open standards that democratize AI technology. These frameworks are designed not only to foster technical interoperability but also to ensure ethical governance and sustainable development within the AI ecosystem.

The Future Trajectory

As we navigate the complexities of AI standardization, it becomes imperative to architect solutions that are not only technically proficient but also scalable and sustainable over time. The establishment of open standards will be critical in ensuring that AI technologies remain adaptive and accessible across various industries.

Conclusion

The path to universal AI adoption is fraught with technical challenges and economic hurdles, but it is navigable through concerted efforts toward standardization and open architecture. By embracing collaborative development and standard-setting, we can unlock a future where AI technologies are as ubiquitous as they are transformative.

There are no bad intentions here. Software developers really do not have any other platform but Git to collaborate. It was only natural that they recommended data scientists to use Git too. Git is a great tool that is optimized to handle a large number

of small, text-based files. On the other hand, AI/ML projects are not just about coding; they involve training an AI model with data. The ML code facilitates the training. And the training requires large datasets. The datasets used for training AI models can be huge and unstructured (images, videos, audio).

There have been attempts to remedy the deficiencies of Git with varying degrees of success. However, I do not think the answer is mending a versioning system that is optimized for code-bases and I do worry about AI/ML development moving towards specialized proprietary platforms instead of using well known open platforms.

Leave a comment

Backstage has emerged as a pivotal platform for crafting developer portals. It’s a universe of its own, bustling with a diverse range of plugins. But here's a twist: as these plugins expand, so does the number of companies that use them. One reality of corporate adoption is the need for security measures. In this post, let's unravel how the Permissions API that is part of the Backstage Core helps secure plugins.

Backstage is a framework for building developer portals with built-in features like software catalogue, templates and plugins. As Backstage's plugin ecosystem flourishes, it brings unparalleled value. Yet, this is accompanied by a crucial responsibility for the plugin providers to ensure security and especially Authorization. The dilemma is the Permissions API, which is the basis for Authorization, is not enabled by default.

Decoding Permissions

In Backstage, plugins reach out to the Permissions API, which in turn consults the Permission Policy. This policy, often provided by vendors or created by in-house platform teams, is where the decisions about who gets to do what are made. Typically, permission policy matches users, groups, roles, attributes, etc., to permissions depending on the implementation and returns an ALLOW, DENY or a maybe to the plugin. Well, two things are alarming in that sentence. 1. there is a maybe decision, which basically expects the plugin to run a conditional and ultimately decide. 2. It is completely left to the plugin to decide to do a permission check and even enforce the decision from the Permission Policy. Unfortunately, this is the reality as of this writing, and the best we can do is to guide the plugin developers to use the Permissions API and use it correctly. Fortunately, securing a Backstage plugin is very well explained in the documentation for backstage.

Incorporating the Permissions API in Backstage plugins is not just about adding layers of security; it's about crafting a seamless developer experience. The combination of well-defined permissions, thorough checks, conditional logic, and front-end integration ensures that plugins remain both functional and secure.

As a plugin developer, how can you decide if your plugin needs to take the Permissions API in use? Here are the criteria that I think you should consider.

• User Roles: Consider the roles and responsibilities of the users of your plugin. If there are diverse user roles with varying levels of access and responsibilities, you may need finer-grained permissions to accommodate these roles effectively.

• Sensitivity of Data: The sensitivity of the data or resource is a key factor. Highly sensitive data, such as personal or financial information, typically requires finer-grained access control.

• Regulatory Compliance: Regulatory requirements, such as GDPR or HIPAA, may dictate the level of granularity for certain types of data access.

• Principle of Least Privilege: Apply the principle of least privilege. Fine-grained permissions can help ensure that users or roles have access only to the specific resources they need to perform their tasks.

Once you decide to use the permissions API and create your own permissions, some best practices to consider are

User Interface: If your plugin has a user interface, design it to present missing actions clearly to users, indicating what options are accessible to them. Do not allow users to click on an action to realize they are not authorized to do the operation.

Separation of Concerns: Ensure that permissions are well-separated concerns and do not unnecessarily mix responsibilities. For example, reading and modifying data should typically be separate permissions.

Testability: Ensure that permissions are testable and that you can verify that users have the correct permissions. Implement tests to verify that access control is functioning as expected.

Documentation and Naming: Document each permission clearly. Use descriptive and consistent naming conventions to make it easy for administrators and developers to understand the purpose of each permission both in terms of its purpose and its impact on the system. Avoid ambiguous or overlapping permissions.

Monitoring and Auditing: Implement mechanisms for monitoring and auditing permissions. It should be possible to track who performs which actions for security monitoring and compliance purposes.

Security Implications: Assess the security implications of your conditions. Be mindful of potential vulnerabilities, such as injection attacks, and implement safeguards to mitigate these risks.

In summary, the expansion of Backstage's plugin ecosystem underscores the necessity of integrating robust security measures into the plugins. This is crucial for maintaining the integrity and reliability of the platform as it grows. Developers are urged to judiciously use the permission API, considering key aspects such as user roles, data sensitivity, and compliance with regulations. By adhering to best practices in design, testing, and documentation, they can ensure that their plugins are not only functional but also secure. Ultimately, applying these principles will bolster Backstage's position as a trusted and versatile platform for developer portals. Failing to do so… well.

ID on IDP stands for internal developer, which clearly identifies the target audience, the developers within an organization. However, the term IDP can be confusing, especially when we get to the letter P. Depending on the context, the P may refer to a platform or portal. So, what exactly is an internal developer platform? Let’s try to bring some order to the chaos by defining both.

Internal Developer Platform

An Internal Developer Platform is a self-service abstraction of all the tools, services, and processes that support an organization's software development. The main goal of an Internal Developer Platform is to ease the cognitive load of software developers. In other words, it is a unified platform that enables developers to focus on building quality software without worrying about infrastructure, security, or deployment.

Internal Developer Portal

Now that we better understand what an internal developer platform is, let's delve into Internal Developer Portal. An internal developer portal is a single pane of glass to an organization's Internal Developer Platform. It puts all the tools, services, and processes, as well as the underlying infrastructure, under one interface.

At a minimum, an Internal Developer Portal is a user interface, hence the name portal. However, the interface can also extend to uniform API access to the developer platform. (imagine a control plane for internal software development). The portal gives developers a single entry point to access all the necessary services to build software. It can also serve as a centralized hub for documentation, tutorials, and best practices.

An Internal Developer Platform is one of the essential tools for modern software development, and an Internal Developer Portal is its essential part. Stay tuned for more posts on IDPs. And if you want to be informed as they are published, use the bottom below to subscribe.

Subscribe now