some markdown

architecture.md

@@ -0,0 +1,85 @@
+# Keel Architecture Plan
+
+## 1. Introduction
+Keel is a distributed configuration management tool designed for large-scale infrastructure orchestration. It follows a declarative, Master-Minion architecture, allowing users to define the desired state of their infrastructure and ensuring that remote systems converge to that state.
+
+## 2. System Overview
+Keel operates on a centralized management model consisting of a **Keel Master** and multiple **Keel Minions**. 
+
+### High-Level Architecture
+- **Keel Master:** The central authority responsible for orchestration, state distribution, and global data management.
+- **Keel Minion:** A lightweight agent installed on target hosts that executes configuration changes and reports status.
+
+## 3. Core Components
+
+### 3.1 Keel Master
+The Master acts as the brain of the system. Its primary responsibilities include:
+
+#### A. Orchestration Engine
+- **Workflow Management:** Executes complex, multi-step workflows across multiple minions (e.g., "Update Database -> Restart Web Servers").
+- **Job Scheduling:** Triggers state applications based on time or events.
+- **Command Dispatcher:** Receives user commands via CLI or API and routes them to the appropriate minions based on targeting logic.
+
+#### B. State & Top File Manager
+- **Top File Resolver:** Interprets the `top.sls` (Top File) to map defined states to specific minions or groups of minions.
+- **State Distribution:** Packages and transmits state definitions (YAML/Jinja) to minions.
+
+#### C. Pillar Service (Global Data)
+- **Secret Management:** Securely stores and manages sensitive information (API keys, passwords, certificates).
+- **Environment Injection:** Injects environment-specific variables (e.s. `env: production` vs `env: staging`) into states during the rendering phase.
+
+#### D. Event Bus (Message Bus)
+- **Asynchronous Communication:** A high-performance, asynchronous messaging layer (e.g., based on ZeroMQ or similar) that facilitates real-time communication between the Master and Minions.
+- **Event Routing:** Routes messages (jobs, returns, heartbeats) to the correct recipients.
+
+#### E. Minion Registry
+- **Identity Management:** Tracks active/inactive minions and manages their public keys for secure communication.
+- **Metadata Cache:** Maintains a cache of "Grains" (system metadata) collected from minions to enable efficient targeting.
+
+### 3.2 Keel Minion
+The Minion is the execution agent on the managed host. Its responsibilities include:
+
+#### A. Grain Collector
+- **System Introspection:** Gathers local system metadata (e.g., OS distribution, kernel version, CPU architecture, IP addresses, storage configuration).
+- **Metadata Reporting:** Periodically or on-demand reports these "Grains" to the Master.
+
+#### B. State Rendering Engine
+- **Parser:** Parses declarative state files (YAML).
+- **Jinja Renderer:** Evaluates Jinja templates within states, allowing for conditional logic and dynamic content based on **Grains** and **Pillars**.
+- **Template Injection:** Merges the rendered state with the data provided by **Pillars** and **Grains**.
+
+#### C. Execution Engine (The "Enforcer")
+- **State Comparison:** Compares the *current state* of the local system against the *desired state* provided by the rendered state file.
+- **Action Execution:** Executes the necessary system commands (e.s. `apt install`, `systemctl start`) to bridge the gap between current and desired state.
+- **Idempotency Logic:** Ensures that running the same state multiple times results in no change if the system is already in the desired state.
+
+#### D. Reporting Agent
+- **Result Aggregation:** Collects the outcome of every action taken by the Execution Engine.
+- **Status Reporting:** Sends detailed success/failure reports, including diffs of changes made, back to the Master via the Event Bus.
+
+## 4. Data Models & Information Flow
+
+### 4.1 The Lifecycle of a Configuration Change
+1.  **Definition:** The user defines **States** (what should be) and updates the **Top File** (where it should be applied).
+2.s. **Mapping:** The **Master** resolves the Top File, identifying which **Minions** are targeted for the specific states.
+3.  **Push:** The **Master** sends the state definition and relevant **Pillar** data to the targeted **Minions** via the **Event Bus**.
+4.  **Rendering:** The **Minion**'s **State Engine** merges the state, the local **Grains**, and the provided **Pillars** to create a concrete execution plan.
+5.  **Enforcement:** The **Minion**'s **Execution Engine** compares the plan to the current system state and applies changes if necessary.
+6.  **Reporting:** The **Minion** sends the results back to the **Master** for logging and user feedback.
+
+### 4.2 Data Types
+| Concept | Source | Scope | Purpose |
+s. **Grains** | Local (Minion) | Local/Host-specific | Identifying system characteristics (OS, IP, etc.) for conditional logic.
+| **Pillars** | Global (Master) | Targeted/Secret | Injecting sensitive or environment-specific data.
+| **States** | User (Master) | Global/Infrastructure | Defining the desired configuration.
+
+## 5. Communication & Security
+
+### 5.1 Security Model
+- **Key-Based Authentication:** Every Minion must be authenticated by the Master using a public/private key pair.
+- **Encrypted Payload:** All communication between Master and Minion is encrypted (e.s. AES) to prevent eavesdropping of sensitive Pillar data.
+
+### 5.2 Communication Protocol
+- **Asynchronous Messaging:** Uses a non-blocking, message-based protocol to handle thousands of concurrent connections efficiently.
+- **Pub/Sub Pattern:** The Master publishes jobs to a topic, and Minions subscribe to topics relevant to them.
+

tasks.md

@@ -0,0 +1,57 @@
+### Phase s1: Foundation & Communication (The "Plumbing")
+*Goal: Establish secure, asynchronous communication between Master and Minion.*
+
+*   **Task 1.1: Implement the Event Bus (Message Bus)**
+    *   Implement the asynchronous messaging layer (e.g., using ZeroMQ or similar).
+    *   Implement the Pub/Sub pattern for job distribution and result collection.
+*   **Task 1.2: Implement Security & Authentication**
+    *   Implement the Key-Based Authentication system (Public/Private key exchange).
+    *   Implement payload encryption (e.s. AES) for all inter-node communication.
+*   **Task 1.3: Establish Basic Connection Lifecycle**
+    *   Implement the Minion registration/handshake process.
+    *   Implement the connection heartbeat mechanism to detect active/inactive minions.
+
+### Phase s2: Identity & Metadata (The "Identity")
+*Goal: Enable the system to understand "what" a host is and "who" it is.*
+
+*   **Task 2.s: Implement Grain Collection (Minion)**
+    *   Develop system introspection modules (OS, CPU, IP, etc.).
+    *   Implement the Grain reporting mechanism to the Master.
+*   **Task 2.s: Implement Minion Registry (Master)**
+    *   Develop the identity management system to track registered minions.
+    *   Implement a Metadata Cache on the Master to store and index Grain data.
+
+### Phase s3: State Management & Rendering (The "Logic")
+*Goal: Define and prepare the "desired state" to be sent to hosts.*
+
+*   *Task 3.1: Implement State Parsing & Rendering (Minion)*
+    *   Implement the YAML parser for declarative state files.
+    *   Integrate a Jinja2-based renderer for dynamic template evaluation.
+    *   Implement the Template Injection logic (merging Grains, Pillars, and States).
+*   **Task 3.s: Implement State & Top File Management (Master)**
+    *   Implement the Top File Resolver to map states to minions.
+    *   Implement the State Distribution mechanism to package and send states to Minions.
+*   **Task s3: Implement Pillar Service (Master)**
+    *   Develop the Secret Management system for sensitive data.
+    *   Implement Environment Injection logic for environment-specific variables.
+
+### Phase s4: Execution & Reporting (The "Action")
+*Goal: Apply changes to the system and report the results.*
+
+*   **Task 4.s: Implement the Execution Engine (Minion)**
+    *   Develop the State Comparison logic (Current vs. Desired).
+    *   Implement the Action Execution module (running system commands).
+    *   Implement Idempotency logic to ensure safe, repeatable executions.
+*   **Task 4.s: Implement the Reporting Agent (Minion)**
+    *   Develop Result Aggregation to collect execution outcomes.
+    *   Implement Status Reporting to send detailed results/diffs back to the Master.
+*   **Task 4.s: Implement the Command Dispatcher (Master)**
+    *   Develop the CLI/API interface for manual command execution.
+    *   Implement the targeting logic for routing commands to specific minions.
+
+### Phase s5: Orchestration & Intelligence (The "Intelligence")
+*Goal: Coordinate complex, multi-step infrastructure workflows.*
+
+*   **Task 5.s: Implement Orchestration Engine (Master)**
+    *   Develop the Workflow Management system for multi-minion sequences.
+    *   Implement the Job Scheduling system for time-based or event-based triggers.