AI Infrastructure

A digital plug or messenger that lets two different software programs talk to each other.

Application Programming Interface. A set of protocols and definitions that allows different software applications to communicate and exchange data. APIs act as the connective tissue of modern software.

Why it matters: Enables AI agents to actually do things by connecting to external tools like Slack, Stripe, or Google.

Full definition: API →

The security guard at the front door of your software that checks IDs and directs traffic.

A server that acts as the single entry point for API requests, routing them to backend services while handling authentication, rate limiting, and logging. Essential in microservices architectures.

Why it matters: Protects backend services from being overwhelmed and ensures secure access to AI models.

Full definition: API Gateway →

Renting powerful computers over the internet instead of buying and keeping them in your own office.

The delivery of computing resources such as servers, storage, databases, and AI services over the internet on a pay-as-you-go basis. Providers including AWS, Azure, and Google Cloud own and maintain the physical hardware, while customers rent capacity on demand and scale it up or down in minutes. Models range from raw infrastructure to fully managed platforms and software.

Why it matters: Cloud computing removed the biggest barrier to AI: access to expensive compute. A startup can now rent supercomputer-grade GPUs by the hour instead of spending millions upfront, which is why nearly all modern AI training and deployment happens in the cloud.

Full definition: Cloud Computing →

Packaging software and all its dependencies into a portable box that runs identically everywhere, from a laptop to the cloud.

A virtualization method that packages applications with their dependencies into isolated containers. Docker and Kubernetes are the dominant tools. Ensures consistent deployment across environments.

Why it matters: The standard deployment method for AI models in production, ensures your model works the same in testing and production.

Full definition: Containerization →

A modern data architecture combining the flexibility of data lakes with the structure of data warehouses, the best of both worlds.

A data management architecture that combines the schema-on-read flexibility of data lakes with the ACID transactions and BI performance of data warehouses. Platforms include Databricks and Snowflake.

Why it matters: Eliminates the need to maintain separate systems for analytics and AI, reducing cost and complexity.

Full definition: Data Lakehouse →

Running AI directly on the device (phone, camera, car) instead of sending data to the cloud, faster and more private.

The deployment of AI models directly on edge devices (smartphones, IoT sensors, vehicles) rather than in centralized cloud servers. Reduces latency, bandwidth costs, and privacy risks.

Why it matters: Critical for real-time applications like autonomous vehicles and industrial robotics where cloud latency is unacceptable.

Full definition: Edge AI →

A centralized library where pre-computed data features are stored and shared across teams and models.

A centralized system for defining, storing, managing, and serving the features that machine learning models consume. It serves the same feature values to both model training and live prediction, maintains a history for reproducibility, and lets multiple teams discover and reuse features instead of rebuilding them. It typically pairs an offline store for training with a low-latency online store for serving.

Why it matters: A feature store prevents training-serving skew, one of the most common and hardest-to-debug production ML bugs, where a model sees differently computed data in production than it trained on. It also accelerates development by turning feature engineering into shared, reusable infrastructure.

Full definition: Feature Store →

Specialized computer chips that can do thousands of math calculations simultaneously, the hardware that makes AI training possible.

Processors originally designed for rendering graphics that are now the primary hardware for training and running AI models. NVIDIA dominates the market. GPU availability is a major bottleneck for AI development.

Why it matters: The most constrained resource in AI, companies that secure GPU capacity have a structural advantage.

Full definition: GPU (Graphics Processing Unit) →

The moment when a trained AI model makes a prediction on new data, using what it learned to answer real questions.

The process of using a trained model to generate predictions on new, unseen data. Inference costs (compute, latency, throughput) are a major consideration for production AI deployments.

Why it matters: Training is a one-time cost; inference runs forever. Optimizing inference costs is often more important than training costs.

Full definition: Inference →

Forcing an AI to respond in clean, structured JSON format instead of free-form text, essential for connecting AI to other software.

A model output setting that constrains the LLM to produce valid JSON in its responses. Ensures programmatic parsability for downstream systems and agent tool calls.

Why it matters: Makes AI outputs machine-readable, critical for agents that need to pass data between tools reliably.

Full definition: JSON Mode →

How long it takes for an AI to respond after you send it a request, the delay between asking and receiving an answer.

The time elapsed between sending a request to an AI system and receiving a response. Measured in milliseconds. Affected by model size, hardware, network distance, and queue depth.

Why it matters: Users abandon interactions after 3 seconds of waiting. For real-time agent actions, latency determines usability.

Full definition: Latency →

The operational practices for deploying, monitoring, and managing large language models in production.

The tools, practices, and processes for operationalizing applications built on large language models. It extends MLOps to the specific challenges of LLMs: managing and versioning prompts, tracking token costs, monitoring latency and output quality, running evaluations to catch regressions, and enforcing safety guardrails. It also covers retrieval pipelines, caching, and fallback logic for when a model is slow or unavailable.

Why it matters: Building an LLM demo is easy; running one reliably and cost-effectively in production is not. LLMOps discipline is what prevents runaway API bills, silent quality drops, and unmonitored failures, and it is what lets a team ship LLM features they can actually trust and maintain.

Full definition: LLMOps →

Building software as a collection of small, independent services that each do one thing well, instead of one giant application.

An architectural pattern where applications are structured as a collection of loosely coupled, independently deployable services. Each service handles a specific business function and communicates via APIs.

Why it matters: The dominant architecture for scalable AI systems, lets you update, scale, and deploy individual components without touching the rest.

Full definition: Microservices →

The practices and tools for reliably deploying, monitoring, and maintaining machine learning models in production.

The discipline of applying DevOps principles to machine learning. Covers model versioning, automated training pipelines, deployment, monitoring, and retraining. Tools include MLflow, Kubeflow, and Weights & Biases.

Why it matters: 87% of ML models never make it to production. MLOps is the bridge between data science experiments and business value.

Full definition: MLOps →

A version-controlled library where all trained models are stored, tracked, and managed, like Git for AI models.

A centralized repository for storing, versioning, and managing trained ML models. Tracks model metadata, performance metrics, and deployment status. Enables reproducibility and governance.

Why it matters: Without a registry, teams lose track of which model is in production, how it was trained, and who approved it.

Full definition: Model Registry →

Shrinking an AI model by reducing the precision of its numbers, making it faster and cheaper to run with minimal quality loss.

A technique for reducing model size and inference cost by representing weights with lower-precision numbers (e.g., 16-bit to 4-bit). Enables running large models on smaller hardware with minimal quality degradation.

Why it matters: Makes it possible to run powerful AI models on consumer hardware and dramatically reduces cloud inference costs.

Full definition: Quantization →

A system's ability to handle growing amounts of work, more users, more data, more requests, without breaking.

The capability of a system to handle increased load by adding resources (horizontal scaling) or upgrading existing resources (vertical scaling). Critical for AI systems that must serve millions of requests.

Why it matters: An AI model that works for 10 users but crashes at 10,000 has no business value, scalability is a requirement, not a feature.

Full definition: Scalability →

How many requests or tasks an AI system can handle per second, its processing speed under real-world conditions.

The rate at which a system processes inputs, measured in requests per second, tokens per second, or tasks per unit time. A key production metric alongside latency and cost.

Why it matters: Determines how many users or tasks your AI system can serve simultaneously, and whether it can handle peak demand.

Full definition: Throughput →

A database designed to store and search AI embeddings, so you can find things by meaning, not just by exact text match.

A database optimized for storing, indexing, and querying high-dimensional vector embeddings. Enables fast similarity search across millions of vectors. Examples include Pinecone, Weaviate, and Chroma.

Why it matters: The essential infrastructure component for RAG systems, semantic search, and AI-powered recommendation engines.

Full definition: Vector Database →