86 lines
5.5 KiB
Markdown
86 lines
5.5 KiB
Markdown
# Keel Architecture Plan
|
|
|
|
## 1. Introduction
|
|
Keel is a distributed configuration management tool designed for large-scale infrastructure orchestration. It follows a declarative, Master-Minion architecture, allowing users to define the desired state of their infrastructure and ensuring that remote systems converge to that state.
|
|
|
|
## 2. System Overview
|
|
Keel operates on a centralized management model consisting of a **Keel Master** and multiple **Keel Minions**.
|
|
|
|
### High-Level Architecture
|
|
- **Keel Master:** The central authority responsible for orchestration, state distribution, and global data management.
|
|
- **Keel Minion:** A lightweight agent installed on target hosts that executes configuration changes and reports status.
|
|
|
|
## 3. Core Components
|
|
|
|
### 3.1 Keel Master
|
|
The Master acts as the brain of the system. Its primary responsibilities include:
|
|
|
|
#### A. Orchestration Engine
|
|
- **Workflow Management:** Executes complex, multi-step workflows across multiple minions (e.g., "Update Database -> Restart Web Servers").
|
|
- **Job Scheduling:** Triggers state applications based on time or events.
|
|
- **Command Dispatcher:** Receives user commands via CLI or API and routes them to the appropriate minions based on targeting logic.
|
|
|
|
#### B. State & Top File Manager
|
|
- **Top File Resolver:** Interprets the `top.sls` (Top File) to map defined states to specific minions or groups of minions.
|
|
- **State Distribution:** Packages and transmits state definitions (YAML/Jinja) to minions.
|
|
|
|
#### C. Pillar Service (Global Data)
|
|
- **Secret Management:** Securely stores and manages sensitive information (API keys, passwords, certificates).
|
|
- **Environment Injection:** Injects environment-specific variables (e.s. `env: production` vs `env: staging`) into states during the rendering phase.
|
|
|
|
#### D. Event Bus (Message Bus)
|
|
- **Asynchronous Communication:** A high-performance, asynchronous messaging layer (e.g., based on ZeroMQ or similar) that facilitates real-time communication between the Master and Minions.
|
|
- **Event Routing:** Routes messages (jobs, returns, heartbeats) to the correct recipients.
|
|
|
|
#### E. Minion Registry
|
|
- **Identity Management:** Tracks active/inactive minions and manages their public keys for secure communication.
|
|
- **Metadata Cache:** Maintains a cache of "Grains" (system metadata) collected from minions to enable efficient targeting.
|
|
|
|
### 3.2 Keel Minion
|
|
The Minion is the execution agent on the managed host. Its responsibilities include:
|
|
|
|
#### A. Grain Collector
|
|
- **System Introspection:** Gathers local system metadata (e.g., OS distribution, kernel version, CPU architecture, IP addresses, storage configuration).
|
|
- **Metadata Reporting:** Periodically or on-demand reports these "Grains" to the Master.
|
|
|
|
#### B. State Rendering Engine
|
|
- **Parser:** Parses declarative state files (YAML).
|
|
- **Jinja Renderer:** Evaluates Jinja templates within states, allowing for conditional logic and dynamic content based on **Grains** and **Pillars**.
|
|
- **Template Injection:** Merges the rendered state with the data provided by **Pillars** and **Grains**.
|
|
|
|
#### C. Execution Engine (The "Enforcer")
|
|
- **State Comparison:** Compares the *current state* of the local system against the *desired state* provided by the rendered state file.
|
|
- **Action Execution:** Executes the necessary system commands (e.s. `apt install`, `systemctl start`) to bridge the gap between current and desired state.
|
|
- **Idempotency Logic:** Ensures that running the same state multiple times results in no change if the system is already in the desired state.
|
|
|
|
#### D. Reporting Agent
|
|
- **Result Aggregation:** Collects the outcome of every action taken by the Execution Engine.
|
|
- **Status Reporting:** Sends detailed success/failure reports, including diffs of changes made, back to the Master via the Event Bus.
|
|
|
|
## 4. Data Models & Information Flow
|
|
|
|
### 4.1 The Lifecycle of a Configuration Change
|
|
1. **Definition:** The user defines **States** (what should be) and updates the **Top File** (where it should be applied).
|
|
2.s. **Mapping:** The **Master** resolves the Top File, identifying which **Minions** are targeted for the specific states.
|
|
3. **Push:** The **Master** sends the state definition and relevant **Pillar** data to the targeted **Minions** via the **Event Bus**.
|
|
4. **Rendering:** The **Minion**'s **State Engine** merges the state, the local **Grains**, and the provided **Pillars** to create a concrete execution plan.
|
|
5. **Enforcement:** The **Minion**'s **Execution Engine** compares the plan to the current system state and applies changes if necessary.
|
|
6. **Reporting:** The **Minion** sends the results back to the **Master** for logging and user feedback.
|
|
|
|
### 4.2 Data Types
|
|
| Concept | Source | Scope | Purpose |
|
|
s. **Grains** | Local (Minion) | Local/Host-specific | Identifying system characteristics (OS, IP, etc.) for conditional logic.
|
|
| **Pillars** | Global (Master) | Targeted/Secret | Injecting sensitive or environment-specific data.
|
|
| **States** | User (Master) | Global/Infrastructure | Defining the desired configuration.
|
|
|
|
## 5. Communication & Security
|
|
|
|
### 5.1 Security Model
|
|
- **Key-Based Authentication:** Every Minion must be authenticated by the Master using a public/private key pair.
|
|
- **Encrypted Payload:** All communication between Master and Minion is encrypted (e.s. AES) to prevent eavesdropping of sensitive Pillar data.
|
|
|
|
### 5.2 Communication Protocol
|
|
- **Asynchronous Messaging:** Uses a non-blocking, message-based protocol to handle thousands of concurrent connections efficiently.
|
|
- **Pub/Sub Pattern:** The Master publishes jobs to a topic, and Minions subscribe to topics relevant to them.
|
|
|