diff --git a/architecture.md b/architecture.md new file mode 100644 index 0000000..25b3b90 --- /dev/null +++ b/architecture.md @@ -0,0 +1,85 @@ +# Keel Architecture Plan + +## 1. Introduction +Keel is a distributed configuration management tool designed for large-scale infrastructure orchestration. It follows a declarative, Master-Minion architecture, allowing users to define the desired state of their infrastructure and ensuring that remote systems converge to that state. + +## 2. System Overview +Keel operates on a centralized management model consisting of a **Keel Master** and multiple **Keel Minions**. + +### High-Level Architecture +- **Keel Master:** The central authority responsible for orchestration, state distribution, and global data management. +- **Keel Minion:** A lightweight agent installed on target hosts that executes configuration changes and reports status. + +## 3. Core Components + +### 3.1 Keel Master +The Master acts as the brain of the system. Its primary responsibilities include: + +#### A. Orchestration Engine +- **Workflow Management:** Executes complex, multi-step workflows across multiple minions (e.g., "Update Database -> Restart Web Servers"). +- **Job Scheduling:** Triggers state applications based on time or events. +- **Command Dispatcher:** Receives user commands via CLI or API and routes them to the appropriate minions based on targeting logic. + +#### B. State & Top File Manager +- **Top File Resolver:** Interprets the `top.sls` (Top File) to map defined states to specific minions or groups of minions. +- **State Distribution:** Packages and transmits state definitions (YAML/Jinja) to minions. + +#### C. Pillar Service (Global Data) +- **Secret Management:** Securely stores and manages sensitive information (API keys, passwords, certificates). +- **Environment Injection:** Injects environment-specific variables (e.s. `env: production` vs `env: staging`) into states during the rendering phase. + +#### D. Event Bus (Message Bus) +- **Asynchronous Communication:** A high-performance, asynchronous messaging layer (e.g., based on ZeroMQ or similar) that facilitates real-time communication between the Master and Minions. +- **Event Routing:** Routes messages (jobs, returns, heartbeats) to the correct recipients. + +#### E. Minion Registry +- **Identity Management:** Tracks active/inactive minions and manages their public keys for secure communication. +- **Metadata Cache:** Maintains a cache of "Grains" (system metadata) collected from minions to enable efficient targeting. + +### 3.2 Keel Minion +The Minion is the execution agent on the managed host. Its responsibilities include: + +#### A. Grain Collector +- **System Introspection:** Gathers local system metadata (e.g., OS distribution, kernel version, CPU architecture, IP addresses, storage configuration). +- **Metadata Reporting:** Periodically or on-demand reports these "Grains" to the Master. + +#### B. State Rendering Engine +- **Parser:** Parses declarative state files (YAML). +- **Jinja Renderer:** Evaluates Jinja templates within states, allowing for conditional logic and dynamic content based on **Grains** and **Pillars**. +- **Template Injection:** Merges the rendered state with the data provided by **Pillars** and **Grains**. + +#### C. Execution Engine (The "Enforcer") +- **State Comparison:** Compares the *current state* of the local system against the *desired state* provided by the rendered state file. +- **Action Execution:** Executes the necessary system commands (e.s. `apt install`, `systemctl start`) to bridge the gap between current and desired state. +- **Idempotency Logic:** Ensures that running the same state multiple times results in no change if the system is already in the desired state. + +#### D. Reporting Agent +- **Result Aggregation:** Collects the outcome of every action taken by the Execution Engine. +- **Status Reporting:** Sends detailed success/failure reports, including diffs of changes made, back to the Master via the Event Bus. + +## 4. Data Models & Information Flow + +### 4.1 The Lifecycle of a Configuration Change +1. **Definition:** The user defines **States** (what should be) and updates the **Top File** (where it should be applied). +2.s. **Mapping:** The **Master** resolves the Top File, identifying which **Minions** are targeted for the specific states. +3. **Push:** The **Master** sends the state definition and relevant **Pillar** data to the targeted **Minions** via the **Event Bus**. +4. **Rendering:** The **Minion**'s **State Engine** merges the state, the local **Grains**, and the provided **Pillars** to create a concrete execution plan. +5. **Enforcement:** The **Minion**'s **Execution Engine** compares the plan to the current system state and applies changes if necessary. +6. **Reporting:** The **Minion** sends the results back to the **Master** for logging and user feedback. + +### 4.2 Data Types +| Concept | Source | Scope | Purpose | +s. **Grains** | Local (Minion) | Local/Host-specific | Identifying system characteristics (OS, IP, etc.) for conditional logic. +| **Pillars** | Global (Master) | Targeted/Secret | Injecting sensitive or environment-specific data. +| **States** | User (Master) | Global/Infrastructure | Defining the desired configuration. + +## 5. Communication & Security + +### 5.1 Security Model +- **Key-Based Authentication:** Every Minion must be authenticated by the Master using a public/private key pair. +- **Encrypted Payload:** All communication between Master and Minion is encrypted (e.s. AES) to prevent eavesdropping of sensitive Pillar data. + +### 5.2 Communication Protocol +- **Asynchronous Messaging:** Uses a non-blocking, message-based protocol to handle thousands of concurrent connections efficiently. +- **Pub/Sub Pattern:** The Master publishes jobs to a topic, and Minions subscribe to topics relevant to them. + diff --git a/tasks.md b/tasks.md new file mode 100644 index 0000000..0fd6c30 --- /dev/null +++ b/tasks.md @@ -0,0 +1,57 @@ +### Phase s1: Foundation & Communication (The "Plumbing") +*Goal: Establish secure, asynchronous communication between Master and Minion.* + +* **Task 1.1: Implement the Event Bus (Message Bus)** + * Implement the asynchronous messaging layer (e.g., using ZeroMQ or similar). + * Implement the Pub/Sub pattern for job distribution and result collection. +* **Task 1.2: Implement Security & Authentication** + * Implement the Key-Based Authentication system (Public/Private key exchange). + * Implement payload encryption (e.s. AES) for all inter-node communication. +* **Task 1.3: Establish Basic Connection Lifecycle** + * Implement the Minion registration/handshake process. + * Implement the connection heartbeat mechanism to detect active/inactive minions. + +### Phase s2: Identity & Metadata (The "Identity") +*Goal: Enable the system to understand "what" a host is and "who" it is.* + +* **Task 2.s: Implement Grain Collection (Minion)** + * Develop system introspection modules (OS, CPU, IP, etc.). + * Implement the Grain reporting mechanism to the Master. +* **Task 2.s: Implement Minion Registry (Master)** + * Develop the identity management system to track registered minions. + * Implement a Metadata Cache on the Master to store and index Grain data. + +### Phase s3: State Management & Rendering (The "Logic") +*Goal: Define and prepare the "desired state" to be sent to hosts.* + +* *Task 3.1: Implement State Parsing & Rendering (Minion)* + * Implement the YAML parser for declarative state files. + * Integrate a Jinja2-based renderer for dynamic template evaluation. + * Implement the Template Injection logic (merging Grains, Pillars, and States). +* **Task 3.s: Implement State & Top File Management (Master)** + * Implement the Top File Resolver to map states to minions. + * Implement the State Distribution mechanism to package and send states to Minions. +* **Task s3: Implement Pillar Service (Master)** + * Develop the Secret Management system for sensitive data. + * Implement Environment Injection logic for environment-specific variables. + +### Phase s4: Execution & Reporting (The "Action") +*Goal: Apply changes to the system and report the results.* + +* **Task 4.s: Implement the Execution Engine (Minion)** + * Develop the State Comparison logic (Current vs. Desired). + * Implement the Action Execution module (running system commands). + * Implement Idempotency logic to ensure safe, repeatable executions. +* **Task 4.s: Implement the Reporting Agent (Minion)** + * Develop Result Aggregation to collect execution outcomes. + * Implement Status Reporting to send detailed results/diffs back to the Master. +* **Task 4.s: Implement the Command Dispatcher (Master)** + * Develop the CLI/API interface for manual command execution. + * Implement the targeting logic for routing commands to specific minions. + +### Phase s5: Orchestration & Intelligence (The "Intelligence") +*Goal: Coordinate complex, multi-step infrastructure workflows.* + +* **Task 5.s: Implement Orchestration Engine (Master)** + * Develop the Workflow Management system for multi-minion sequences. + * Implement the Job Scheduling system for time-based or event-based triggers. \ No newline at end of file