From 8166e9b345674834b1a35c69d5719d5bbaeb8240 Mon Sep 17 00:00:00 2001 From: Kat Morgan Date: Sun, 17 Nov 2024 21:25:19 -0800 Subject: [PATCH] documentation refactor --- docs/DOCUMENTATION.md | 163 ---- docs/DOCUMENTATION_PLAN.md | 319 -------- docs/HOW_TO.md | 260 ------ docs/PULUMI_PYTHON.md | 295 ------- docs/README.md | 90 --- docs/call_to_action.md | 248 ------ docs/compliance/COMPLIANCE.md | 498 ------------ docs/developer_guide/README.md | 694 +++++++++++----- docs/developer_guide/ROADMAP.md | 298 +++++++ .../contribution_guidelines.md | 378 --------- docs/{ => developer_guide}/devcontainer.md | 0 docs/developer_guide/documentation.md | 434 ++++++++++ .../konductor_developer_guide.md | 738 ------------------ docs/developer_guide/module_refactoring.md | 229 ------ docs/developer_guide/modules/core/README.md | 351 --------- docs/developer_guide/pulumi_python.md | 601 ++++++++++++++ .../{modules => roadmaps}/aws/README.md | 0 .../aws/developer_guide.md | 0 .../aws/eks_donor_template.md | 0 .../aws/eks_opentelemetry_docs.md | 0 .../aws/implementation_roadmap.md | 0 docs/getting_started.md | 165 ---- docs/infrastructure_as_code.md | 131 ---- docs/modules/core/README.md | 466 +++++++++++ docs/pulumi_iac.md | 168 ---- docs/reference/TypeDict.md | 368 --------- docs/reference/best_practices.md | 211 ----- docs/reference/pulumi_python.md | 475 ----------- docs/reference/style_guide.md | 267 ------- docs/roadmaps/ROADMAP.md | 218 ------ docs/roadmaps/ROADMAP_Addendum.md | 417 ---------- 31 files changed, 2292 insertions(+), 6190 deletions(-) delete mode 100644 docs/DOCUMENTATION.md delete mode 100644 docs/DOCUMENTATION_PLAN.md delete mode 100644 docs/HOW_TO.md delete mode 100644 docs/PULUMI_PYTHON.md delete mode 100644 docs/README.md delete mode 100644 docs/call_to_action.md delete mode 100644 docs/compliance/COMPLIANCE.md create mode 100644 docs/developer_guide/ROADMAP.md delete mode 100644 docs/developer_guide/contribution_guidelines.md rename docs/{ => developer_guide}/devcontainer.md (100%) create mode 100644 docs/developer_guide/documentation.md delete mode 100644 docs/developer_guide/konductor_developer_guide.md delete mode 100644 docs/developer_guide/module_refactoring.md delete mode 100644 docs/developer_guide/modules/core/README.md create mode 100644 docs/developer_guide/pulumi_python.md rename docs/developer_guide/{modules => roadmaps}/aws/README.md (100%) rename docs/developer_guide/{modules => roadmaps}/aws/developer_guide.md (100%) rename docs/developer_guide/{modules => roadmaps}/aws/eks_donor_template.md (100%) rename docs/developer_guide/{modules => roadmaps}/aws/eks_opentelemetry_docs.md (100%) rename docs/developer_guide/{modules => roadmaps}/aws/implementation_roadmap.md (100%) delete mode 100644 docs/getting_started.md delete mode 100644 docs/infrastructure_as_code.md create mode 100644 docs/modules/core/README.md delete mode 100644 docs/pulumi_iac.md delete mode 100644 docs/reference/TypeDict.md delete mode 100644 docs/reference/best_practices.md delete mode 100644 docs/reference/pulumi_python.md delete mode 100644 docs/reference/style_guide.md delete mode 100644 docs/roadmaps/ROADMAP.md delete mode 100644 docs/roadmaps/ROADMAP_Addendum.md diff --git a/docs/DOCUMENTATION.md b/docs/DOCUMENTATION.md deleted file mode 100644 index fbaf8c6..0000000 --- a/docs/DOCUMENTATION.md +++ /dev/null @@ -1,163 +0,0 @@ -# Documentation Guidelines for Module Maintainers and Documentation Developers - -## Purpose - -This document provides comprehensive guidelines for module maintainers and documentation developers contributing to the **Konductor Infrastructure as Code (IaC) project**. The goal is to produce high-quality, informative, and accessible documentation that serves a diverse audience—from novice homelab enthusiasts to senior DevOps professionals and principal platform engineers. - -By adhering to these guidelines, you can ensure that all documentation is consistent, comprehensive, and aligned with industry best practices, enhancing its value to developers, users, and practitioners. - ---- - -## General Principles - -### 1. **Alignment with Project Standards** - -- **Consistency**: Ensure all documentation aligns with the project's established standards and guidelines. -- **Best Practices**: Incorporate documentation best practices from leading cloud-native projects like Kubernetes, Docker, and others. - -### 2. **Audience Awareness** - -- **Inclusivity**: Write documentation that is accessible and informative for a wide range of readers, from beginners to experts. -- **Clarity**: Use clear, concise language and avoid unnecessary jargon. When technical terms are necessary, provide definitions or explanations. -- **Depth and Detail**: Provide sufficient detail to help advanced users while ensuring that beginners can understand and follow along. - -### 3. **Content Quality** - -- **Accuracy**: Ensure all information is accurate and up-to-date with the latest project developments. -- **Completeness**: Cover all necessary topics, including setup instructions, usage examples, troubleshooting, and best practices. -- **Relevance**: Focus on content that adds value to the user experience and aids in the understanding of the module or feature. - -### 4. **Structural Organization** - -- **Logical Flow**: Organize content in a logical sequence, starting from general concepts and progressing to specific details. -- **Predictable Layout**: Use a consistent structure across all documents to help users know where to find information. -- **Modularity**: Break down documentation into manageable sections or documents, each focusing on a specific aspect or feature. - -### 5. **Stylistic Consistency** - -- **Formatting Standards**: Adhere to consistent formatting throughout all documents, including headings, code blocks, lists, and emphasis. -- **Tone and Voice**: Maintain a professional and approachable tone. Encourage engagement and learning by being supportive and inclusive. -- **Terminology**: Use consistent terminology across all documents to prevent confusion. - ---- - -## Guidelines for Authoring and Revising Documentation - -### A. **Document Structure and Sections** - -- **Introduction**: Begin with an introduction that outlines the purpose, scope, and audience of the document. -- **Table of Contents**: Provide a table of contents for easy navigation, especially for longer documents. -- **Sections and Subsections**: Organize content into clear sections with descriptive headings. -- **Conclusion and Next Steps**: End with a summary of key points and suggestions for further reading or actions. - -### B. **Content Development** - -- **Use Case Examples**: Include practical examples and real-world use cases to illustrate concepts. -- **Code Samples**: Provide code snippets where applicable, ensuring they are tested and functional. -- **Visual Aids**: Use diagrams, charts, or tables to explain complex ideas or workflows. -- **FAQs and Troubleshooting**: Anticipate common questions or issues and address them proactively. - -### C. **Style and Formatting** - -- **Markdown Standards**: Use proper Markdown syntax for formatting documents, including appropriate heading levels, bold and italics, code blocks, and lists. -- **Code Blocks**: Use fenced code blocks with language identifiers for syntax highlighting (e.g., ```python). -- **Inline Code and Commands**: Use backticks for inline code references or commands (e.g., `kubectl get pods`). -- **Links and References**: Include hyperlinks to relevant sections, documents, or external resources. - -### D. **Clarity and Accessibility** - -- **Plain Language**: Write in plain language to make content accessible to non-native English speakers and those less familiar with the subject matter. -- **Definitions and Glossaries**: Provide definitions for specialized terms and consider including a glossary for complex topics. -- **Accessibility Standards**: Follow accessibility guidelines, such as using alt text for images and ensuring sufficient color contrast. - -### E. **Review and Quality Assurance** - -- **Proofreading**: Carefully proofread documents to correct grammatical errors, typos, and inconsistencies. -- **Peer Review**: Engage other team members to review documentation for accuracy, clarity, and completeness. -- **Continuous Updates**: Regularly update documentation to reflect changes in the project, deprecations, or new features. - ---- - -## Module Maintainers Specific Guidelines - -### A. **Module Documentation Structure** - -- **README.md**: Provide an overview of the module, including its purpose, features, and how it fits into the larger project. -- **Installation Guide**: Include step-by-step instructions on how to install and configure the module. -- **Usage Instructions**: Offer detailed usage examples, including common use cases and advanced configurations. -- **API References**: If applicable, provide API documentation with explanations of available functions, classes, or methods. -- **Changelog**: Maintain a changelog that documents significant changes, enhancements, and bug fixes. - -### B. **Consistency Across Modules** - -- **Standardized Templates**: Use standardized templates for module documentation to ensure consistency. -- **Naming Conventions**: Follow consistent naming conventions for files, directories, and headings. -- **Cross-Module References**: Reference related modules where appropriate and explain how they interact. - -### C. **Versioning and Compatibility** - -- **Version Information**: Clearly indicate the module version and compatible versions of dependencies. -- **Deprecation Notices**: Provide advance notice of deprecated features and guidance on migration paths. - -### D. **Contribution Guidelines** - -- **How to Contribute**: Include clear instructions for contributing to the module, such as coding standards, testing requirements, and submission processes. -- **Issue Reporting**: Explain how users can report bugs or request features, including any templates or guidelines to follow. - ---- - -## Documentation Developers Specific Guidelines - -### A. **Documentation Contribution Process** - -- **Style Guide Compliance**: Familiarize yourself with and adhere to the project's documentation style guide. -- **Documentation Planning**: Collaborate with developers and maintainers to plan documentation updates alongside code changes. -- **Use of Tools**: Utilize documentation tools and generators where appropriate (e.g., Sphinx, MkDocs). - -### B. **Content Maintenance** - -- **Content Audits**: Regularly review existing documentation to identify outdated information or gaps. -- **User Feedback**: Incorporate feedback from users to improve documentation clarity and usefulness. -- **Localization**: If applicable, support localization efforts by preparing documentation for translation. - -### C. **Collaboration with Development Teams** - -- **Integration with Development Workflows**: Align documentation updates with development cycles and release schedules. -- **Documentation in Pull Requests**: Encourage the inclusion of documentation changes in code pull requests when features are added or modified. - ---- - -## Drawing Inspiration from Leading Projects - -To meet or exceed the standards of mainstream cloud-native projects like Kubernetes, consider the following practices: - -- **Adopt a Documentation Style Guide**: Use established style guides like the [Kubernetes Documentation Style Guide](https://kubernetes.io/docs/contribute/style/style-guide/) as a reference. -- **Use Structured Formats**: Implement documentation structures similar to those found in Kubernetes or Docker, which often include concepts, tasks, tutorials, and reference sections. -- **Implement Documentation Tests**: Utilize tools that can test the validity of links, code snippets, and formatting within documentation. -- **Community Engagement**: Foster a community around documentation by recognizing contributors and encouraging participation through documentation sprints or hackathons. - ---- - -## Additional Considerations - -- **Accessibility Compliance**: Ensure documentation meets accessibility standards, such as WCAG 2.1, to make content usable by everyone. -- **Searchability**: Optimize documentation for search engines and include metadata to improve discoverability. -- **Documentation Metrics**: Track metrics like page views, time on page, and user feedback to assess the effectiveness of documentation and identify areas for improvement. -- **Disaster Recovery and Archiving**: Implement processes for backing up documentation and maintaining version history. - ---- - -## Conclusion - -By following these guidelines, module maintainers and documentation developers can create high-quality documentation that enhances the Konductor IaC project's usability and adoption. Consistent, clear, and comprehensive documentation is essential for empowering users and fostering a collaborative community. - -Your efforts are crucial to the project's success, and your contributions are highly valued. Together, we can build an exceptional knowledge base that serves the needs of all users and contributors. - ---- - -## Getting Help - -If you have questions or need assistance with documentation, please reach out through the following channels: - -- **Discord Channel**: Join our project's Discord channel for real-time discussions. -- **Issue Tracker**: Open an issue in the project's repository with the label `documentation`. diff --git a/docs/DOCUMENTATION_PLAN.md b/docs/DOCUMENTATION_PLAN.md deleted file mode 100644 index 7458f60..0000000 --- a/docs/DOCUMENTATION_PLAN.md +++ /dev/null @@ -1,319 +0,0 @@ -# Documentation Analysis and Reorganization Plan - -## Introduction - -This analysis examines the current state of the documentation after recent refactoring efforts. Our goal is to identify issues across various dimensions—functional, informational, architectural, stylistic, compliance, context, cohesion, and omissions. Based on this analysis, we propose a reorganized documentation structure that enhances reliability, intuitiveness, and accessibility for maintainers, developers, and end users. - -By aligning with the project's documentation guidelines, we aim to produce high-quality, informative, and accessible documentation that serves a diverse audience. - ---- - -## Current Documentation Analysis - -### Overview of Existing Documents - -1. **PULUMI_PYTHON.md** - - **Purpose**: Outlines development practices, techniques, and requirements for working on the Pulumi project. - - **Content**: Project setup, dependency management with Poetry, enforcing type checking with Pyright, using `TypedDict`, best practices, and code standards. - -2. **Konductor User Guide** - - **Purpose**: Provides an in-depth overview of the design principles, code structure, and best practices for module development within the Konductor IaC codebase. - - **Content**: Introduction, design principles, code structure, configuration management, module development guide, example module (Cert Manager), conclusion, and next steps. - -3. **Konductor Developer Guide** - - **Purpose**: Intended for developers contributing to the Konductor IaC codebase. - - **Content**: Code structure, development best practices, contribution workflow, adding enhancements, testing and validation, documentation standards, support, and resources. - -4. **AWS Module Implementation Roadmap** - - **Purpose**: Provides a comprehensive guide for implementing a scalable and modular AWS infrastructure using Pulumi and Python. - - **Content**: Introduction, goals and objectives, code structure, configuration management, AWS organization setup, IAM management, deploying workloads, secrets management, main execution flow, testing and validation, documentation best practices, conclusion, and additional resources. - -5. **Module-Specific Documents** - - **eks_donor_template.md** - - **Purpose**: Detailed guide and code walkthrough for setting up an Amazon EKS cluster with supporting AWS infrastructure. - - **eks_opentelemetry_docs.md** - - **Purpose**: Documentation on integrating AWS Distro for OpenTelemetry (ADOT) with Amazon EKS. - -6. **Other Documents** - - **CALL_TO_ACTION.md**: Purpose unclear without content. - - **COMPLIANCE.md**: Likely covers compliance requirements and standards. - - **TypeDict.md**: Possibly explains the usage of `TypedDict` in the codebase. - - **ROADMAP.md** and **ROADMAP_Addendum.md**: Provide project roadmaps and additional planning details. - - **Multiple README.md and DEVELOPER.md Files**: Present in various directories, potentially leading to confusion. - -### Identified Issues - -#### Functional - -- **Redundancy and Overlap**: Multiple `README.md` and `DEVELOPER.md` files scattered across directories can cause confusion. -- **Scattered Information**: Important information is spread across different locations, making it difficult to find. -- **Clarity of Purpose**: Some documents lack clear descriptions of their intended audience or purpose. - -#### Informational - -- **Duplication**: Similar content may be repeated in different documents. -- **Gaps**: Some advanced topics, FAQs, or troubleshooting guides are missing. - -#### Architectural - -- **Inconsistent Organization**: Documentation is not centralized, leading to a fragmented experience. -- **Predictability**: Users may not know where to look for specific information due to inconsistent placement of documents. - -#### Stylistic - -- **Inconsistency**: Variations in writing style, formatting, and terminology across documents. -- **Formatting Issues**: Lack of standardized formatting guidelines may affect readability. - -#### Compliance - -- **Adherence to Standards**: Some documents may not fully align with the standards outlined in `PULUMI_PYTHON.md`. - -#### Context - -- **Audience Targeting**: Unclear distinctions between documents intended for end users, developers, or maintainers. -- **Assumed Knowledge**: Some documents may assume prior knowledge not shared by all readers. - -#### Cohesion - -- **Disconnected Sections**: Lack of cross-referencing between related documents. -- **Flow**: The progression from introductory to advanced topics may not be logical. - -#### Omissions - -- **Central Index or Overview**: Absence of a central document that maps out the entire documentation structure. -- **Contribution Templates**: Missing templates or examples for contributions, issues, or pull requests. -- **FAQs and Troubleshooting**: Lack of dedicated sections for common issues and their resolutions. -- **Accessibility Compliance**: No mention of accessibility standards. - ---- - -## Proposed Documentation Reorganization - -To address the identified issues and align with the documentation guidelines, we'll reorganize the documentation into a centralized and predictable structure. This new layout enhances accessibility, reduces redundancy, and improves the overall user experience. - -### New Documentation Structure - -``` -docs/ -├── README.md -├── getting_started.md -├── user_guide/ -│ ├── README.md -│ ├── konductor_user_guide.md -│ └── faq_and_troubleshooting.md -├── developer_guide/ -│ ├── README.md -│ ├── konductor_developer_guide.md -│ ├── contribution_guidelines.md -│ └── modules/ -│ ├── aws/ -│ │ ├── README.md -│ │ ├── implementation_roadmap.md -│ │ ├── developer_guide.md -│ │ ├── eks_donor_template.md -│ │ ├── eks_opentelemetry_docs.md -│ │ └── changelog.md -│ └── cert_manager/ -│ ├── README.md -│ ├── developer_guide.md -│ └── changelog.md -├── modules/ -│ ├── aws/ -│ │ ├── README.md -│ │ ├── usage_guide.md -│ │ ├── installation_guide.md -│ │ └── faq_and_troubleshooting.md -│ └── cert_manager/ -│ ├── README.md -│ ├── usage_guide.md -│ ├── installation_guide.md -│ └── faq_and_troubleshooting.md -├── reference/ -│ ├── PULUMI_PYTHON.md -│ ├── TypedDict.md -│ └── style_guide.md -├── compliance/ -│ └── COMPLIANCE.md -├── roadmaps/ -│ ├── ROADMAP.md -│ └── ROADMAP_Addendum.md -├── contribution_templates/ -│ ├── issue_template.md -│ ├── pull_request_template.md -│ └── feature_request_template.md -├── call_to_action.md -``` - -### Explanation of the New Structure - -#### 1. **docs/** (Root Documentation Directory) - -- **Purpose**: Centralizes all documentation, making it the go-to place for any information related to the project. -- **Contents**: - - `README.md`: Provides an overview of the documentation structure, accessibility considerations, and guides users to specific sections. - - `getting_started.md`: A quick-start guide for new users to set up and begin using the project. - -#### 2. **user_guide/** - -- **Purpose**: Contains all user-focused documentation, helping end users understand how to use the project. -- **Contents**: - - `README.md`: Introduces the user guides and provides a table of contents. - - `konductor_user_guide.md`: The main user guide for the Konductor IaC platform. - - `faq_and_troubleshooting.md`: Addresses common questions and issues. - -#### 3. **developer_guide/** - -- **Purpose**: Houses developer-focused documentation, including guidelines for contributing and extending the project. -- **Contents**: - - `README.md`: Overview of the developer guides. - - `konductor_developer_guide.md`: Detailed guide for developers working on the Konductor codebase. - - `contribution_guidelines.md`: Centralized document outlining the contribution workflow, including templates. - - `modules/`: Subdirectory containing module-specific developer guides. - -#### 4. **developer_guide/modules/** - -- **Purpose**: Organizes developer documentation for individual modules. -- **Contents**: - - **aws/**: - - `README.md`: Introduction to the AWS module. - - `implementation_roadmap.md`: The AWS Module Implementation Roadmap. - - `developer_guide.md`: Developer guide specific to the AWS module. - - `eks_donor_template.md`: Documentation for the EKS donor template. - - `eks_opentelemetry_docs.md`: Guide on integrating ADOT with EKS. - - `changelog.md`: Documents significant changes, enhancements, and bug fixes. - - **cert_manager/**: - - `README.md`: Introduction to the Cert Manager module. - - `developer_guide.md`: Developer guide specific to the Cert Manager module. - - `changelog.md`: Documents significant changes, enhancements, and bug fixes. - -#### 5. **modules/** - -- **Purpose**: Contains user-facing documentation for individual modules. -- **Contents**: - - **aws/**: - - `README.md`: Overview and basic usage of the AWS module. - - `installation_guide.md`: Step-by-step instructions on how to install and configure the module. - - `usage_guide.md`: Detailed instructions on using the AWS module. - - `faq_and_troubleshooting.md`: Addresses common questions and issues. - - **cert_manager/**: - - `README.md`: Overview and basic usage of the Cert Manager module. - - `installation_guide.md`: Step-by-step instructions on how to install and configure the module. - - `usage_guide.md`: Detailed instructions on using the Cert Manager module. - - `faq_and_troubleshooting.md`: Addresses common questions and issues. - -#### 6. **reference/** - -- **Purpose**: Stores reference materials and standards applicable across the project. -- **Contents**: - - `PULUMI_PYTHON.md`: Pulumi Python development standards. - - `TypedDict.md`: Explanation and usage guidelines for `TypedDict`. - - `style_guide.md`: Documentation style guide outlining formatting, tone, and terminology standards. - -#### 7. **compliance/** - -- **Purpose**: Contains compliance requirements and related documentation. -- **Contents**: - - `COMPLIANCE.md`: Details on compliance standards and how the project adheres to them. - -#### 8. **roadmaps/** - -- **Purpose**: Provides planning documents and future development roadmaps. -- **Contents**: - - `ROADMAP.md`: The overall project roadmap. - - `ROADMAP_Addendum.md`: Additional details or updates to the roadmap. - -#### 9. **contribution_templates/** - -- **Purpose**: Contains templates for issues, pull requests, and feature requests to standardize contributions. -- **Contents**: - - `issue_template.md`: Template for reporting issues. - - `pull_request_template.md`: Template for submitting pull requests. - - `feature_request_template.md`: Template for proposing new features. - -#### 10. **call_to_action.md** - -- **Purpose**: A document highlighting ways for the community to contribute or engage with the project. - -### Benefits of the New Structure - -- **Centralization**: All documentation is located under the `docs/` directory, making it easy to find. -- **Predictability**: Users can intuitively navigate to the appropriate section based on their needs. -- **Audience Clarity**: Clear separation between user guides and developer guides ensures that each audience can find relevant information quickly. -- **Reduced Redundancy**: Consolidates duplicate documents and organizes content logically. -- **Consistency**: Enables uniform formatting, style, and terminology across all documents by including a `style_guide.md`. -- **Improved Navigation**: README files in each directory provide overviews and link to sub-documents. -- **Enhanced Cohesion**: Related documents are grouped together, facilitating a logical flow of information. -- **Ease of Maintenance**: A structured layout simplifies updates and the addition of new documentation. -- **Accessibility Compliance**: By including accessibility considerations in the `docs/README.md` and following guidelines in the `style_guide.md`, we ensure documentation is accessible to all users. - ---- - -## Implementation Plan - -To transition to the new documentation structure without losing any content or value, we propose the following steps: - -1. **Create the `docs/` Directory**: - - Move all existing documentation files into the `docs/` directory. - - Update any relative links within documents to reflect the new paths. - -2. **Organize Documentation into Subdirectories**: - - Categorize documents based on their audience and purpose. - - For example, move `Konductor User Guide` into `docs/user_guide/konductor_user_guide.md`. - -3. **Consolidate Duplicate Documents**: - - Merge multiple `README.md` and `DEVELOPER.md` files where appropriate. - - Ensure that module-specific guides are placed in their respective directories under `developer_guide/modules/`. - -4. **Standardize Document Formatting**: - - Apply consistent styling, headings, and formatting across all documents according to the `style_guide.md`. - - Use a markdown linter or formatter to enforce style guidelines. - -5. **Update the Root `README.md`**: - - Provide an introduction to the project. - - Include links to the main sections of the documentation under `docs/`. - - Highlight accessibility features and compliance. - -6. **Create Indexes and Tables of Contents**: - - In `docs/README.md`, include a high-level table of contents for the entire documentation. - - In each subdirectory's `README.md`, provide an overview of the contents and purpose. - -7. **Add Missing Sections**: - - **FAQs and Troubleshooting**: Create dedicated sections to address common issues. - - **Accessibility Compliance**: Ensure documents meet accessibility standards, as outlined in `DOCUMENTATION.md`. - - **Glossary**: Consider adding a glossary for complex terms. - -8. **Review for Omissions and Gaps**: - - Ensure that all critical topics are covered. - - Incorporate user feedback to identify areas needing improvement. - -9. **Ensure Compliance with Standards**: - - Cross-reference documents with `PULUMI_PYTHON.md` and `style_guide.md` to ensure adherence to coding and documentation standards. - -10. **Update Contribution Guidelines**: - - In `docs/developer_guide/contribution_guidelines.md`, outline the new documentation structure. - - Provide instructions for adding or updating documentation within the new layout. - - Include templates from `contribution_templates/` for consistency. - -11. **Implement Accessibility and Searchability Enhancements**: - - Add metadata to documents to improve search engine optimization (SEO). - - Use headings and alt text appropriately for accessibility. - -12. **Communicate the Changes**: - - Inform the team and community about the reorganization through the `call_to_action.md`. - - Update any external links or references to documentation. - -13. **Set Up Documentation Metrics**: - - Implement tools to track documentation usage and effectiveness. - - Use metrics to guide future improvements. - -14. **Backup and Version Control**: - - Ensure all documentation changes are committed to version control. - - Implement a backup strategy for documentation. - ---- - -## Conclusion - -By reorganizing the documentation into a centralized and predictable structure, we enhance the usability and accessibility of information for all stakeholders. This new layout addresses the issues identified in the analysis by providing clear pathways to information, reducing redundancy, and ensuring consistency across all documents. - -Maintainers, developers, and end users will benefit from the improved organization, making it easier to find the information they need and contribute effectively to the project. This reorganization sets a strong foundation for future growth and scalability of the documentation as the project evolves. diff --git a/docs/HOW_TO.md b/docs/HOW_TO.md deleted file mode 100644 index 5bbe120..0000000 --- a/docs/HOW_TO.md +++ /dev/null @@ -1,260 +0,0 @@ -# Konductor DevOps Template - -## Introduction - -Welcome to the Konductor DevOps Template. - -This repository includes baseline dependencies and boilerplate artifacts for operating and developing cloud infrastructure automation. - -Follow the steps below to configure AWS credentials, set up your development environment, and verify access to AWS and EKS resources. - -## Prerequisites - -Before you begin, ensure you have the following installed: - -- **Python**: Version 3.8 or higher. -- **Poetry**: For dependency management and packaging. [Install Poetry](https://python-poetry.org/docs/#installation). -- **AWS CLI**: Version 2.x or higher. -- **Pulumi CLI**: For managing infrastructure as code. -- **Git**: For cloning repositories. -- **Kubectl**: For interacting with Kubernetes clusters. -- **sudo**: For executing administrative commands. - -> **Note:** All dependencies are automatically supplied in the [ghcr.io/containercraft/devcontainer](https://github.com/containercraft/devcontainer) image powering the VSCode Dev Container included in this repository by the [.devcontainer/devcontainer.json](.devcontainer/devcontainer.json) and [.devcontainer/Dockerfile](.devcontainer/Dockerfile). - -## Steps to Recreate - -Follow the steps below to set up your environment: - -### 1. Initialize the Development Environment - -#### a. Clone the Repository - -Clone this repository to your local machine: - -```bash -git clone https://github.com/containercraft/konductor.git -cd konductor -``` - -#### b. Install Dependencies with Poetry - -We use [Poetry](https://python-poetry.org/) for dependency management and packaging. Poetry ensures that our development environment is consistent, dependencies are properly managed, and collaboration is streamlined. - -Install the project dependencies: - -```bash -poetry install -``` - -This command will create a virtual environment and install all dependencies specified in `pyproject.toml`. - -#### c. Activate the Virtual Environment - -Activate the virtual environment: - -```bash -poetry shell -``` - -Alternatively, you can prefix commands with `poetry run`. - -#### d. Configure Pulumi to Use Poetry - -Ensure that `Pulumi.yaml` specifies Poetry as the toolchain: - -```yaml -name: your-pulumi-project -runtime: - name: python - options: - toolchain: poetry -``` - -#### e. Install Pulumi Dependencies - -Install Pulumi dependencies: - -```bash -pulumi install -``` - -This command ensures that Pulumi recognizes and utilizes the Poetry-managed environment. - -### 2. Enforce Type Checking with Pyright - -Type checking enhances code reliability and maintainability. We enforce strict type checking using [Pyright](https://github.com/microsoft/pyright). - -#### a. Verify Type Checking - -Run Pyright to check for type errors: - -```bash -poetry run pyright -``` - -Ensure that there are no type errors before proceeding. If type errors are detected, fix them according to the standards outlined in [PULUMI_PYTHON.md](../docs/PULUMI_PYTHON.md). - -### 3. Authenticate with Pulumi - -Login to Pulumi Cloud: - -```bash -pulumi login -``` - -### 4. Load Environment Variables and AWS Credentials - -Use Pulumi to load environment variables, configuration files, and credentials. - -*Note:* Replace ``, ``, and `` with your Pulumi organization, project name, and stack name. - -```bash -export ENVIRONMENT="//" -eval $(pulumi env open --format=shell $ENVIRONMENT | tee .tmpenv; direnv allow) -echo "Loaded environment $ENVIRONMENT" - -alias aws='aws --profile smdc-cba' -``` - -### 5. Validate AWS CLI Access - -Get Caller Identity to verify your AWS identity: - -```bash -aws --profile smdc-cba sts get-caller-identity -``` - -### 6. Deploy Infrastructure as Code (IaC) - -Before deploying, ensure that your code passes type checking to maintain code quality. - -#### a. Run Type Checking - -```bash -poetry run pyright -``` - -If type errors are detected, the deployment will halt, and errors will be displayed. - -#### b. Deploy with Pulumi - -Deploy the infrastructure using Pulumi: - -```bash -pulumi up --yes --stack // --skip-preview --refresh -``` - -Replace `//` with your specific Pulumi stack information. - -### 7. Install the SMCE CLI Tool - -Clone the SMCE CLI repository and set up the `smce` CLI: - -```bash -cd ~ -rm -rf ~/smce-cli ~/.local/bin/smce -ln -sf $GIT_CONFIG ~/.gitconfig - -git clone https://git.smce.nasa.gov/smce-administration/smce-cli.git ~/smce-cli -cd ~/smce-cli - -mkdir -p ~/.local/bin -cp -f ~/smce-cli/smce ~/.local/bin/smce -chmod +x ~/.local/bin/smce - -smce --help || true -``` - -### 8. Configure AWS and Kubernetes Using SMCE CLI - -Set up AWS Multi-Factor Authentication (MFA): - -> **Note:** Enhance `smce-cli` to auto-export MFA environment variables. - -```bash -smce awsconfig mfa -``` - -### 9. Test AWS CLI Access - -List S3 buckets to confirm AWS access and verify your AWS identity: - -```bash -aws s3 ls -aws sts get-caller-identity -``` - -### 10. Update Kubernetes Configuration for EKS Cluster - -Update your kubeconfig file to interact with your EKS cluster: - -```bash -aws eks update-kubeconfig --profile main --region us-east-1 --name smce-gitops -``` - -Generate a new Kubernetes configuration: - -```bash -smce kubeconfig generate -``` - -Generate an authentication token for the EKS cluster: - -```bash -aws eks get-token --region us-east-1 --cluster-name smce-gitops --output json -``` - -Replace `` with your MFA device's ARN and provide your MFA token code in place of `$MFA_TOKEN`. - -### 11. Configure Kubectl Alias and Verify Kubernetes Access - -List available Kubernetes contexts: - -```bash -kubectl --kubeconfig ~/.kube/smce config get-contexts -``` - -Retrieve the list of nodes in your Kubernetes cluster: - -```bash -kubectl --kubeconfig ~/.kube/smce get nodes -``` - -Check the Kubernetes client and server versions with verbose output: - -```bash -kubectl version -v=8 -``` - -## Conclusion - -By following these steps, you've set up your environment to interact with AWS services and your EKS cluster. This setup is essential for deploying and managing applications using the Konductor DevOps Template. - -## Resources - -- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) -- [Pulumi Documentation](https://www.pulumi.com/docs/) -- [SMCE CLI Repository](https://git.smce.nasa.gov/smce-administration/smce-cli) -- [Kubernetes Documentation](https://kubernetes.io/docs/home/) -- [Poetry Documentation](https://python-poetry.org/docs/) -- [Pyright Documentation](https://github.com/microsoft/pyright) - -## Troubleshooting - -**Note:** If you encounter authentication issues due to MFA requirements, test temporary session credentials using the following command: - -```bash -aws sts get-session-token \ - --duration-seconds 129600 \ - --profile default \ - --serial-number \ - --token-code $MFA_TOKEN -``` - -## Bonus: Launch Kubernetes in Docker - -```bash -cd .. -task kubernetes -``` diff --git a/docs/PULUMI_PYTHON.md b/docs/PULUMI_PYTHON.md deleted file mode 100644 index ab1546d..0000000 --- a/docs/PULUMI_PYTHON.md +++ /dev/null @@ -1,295 +0,0 @@ -# Developer Documentation for Pulumi Projects - -This document outlines the development practices, techniques, and requirements for working on our advanced platform engineering Pulumi project. It focuses on ensuring code quality, maintainability, and adherence to modern Python and Pulumi best practices. - -## Table of Contents - -- [Project Setup](#project-setup) - - [Dependency Management with Poetry](#dependency-management-with-poetry) - - [Initializing the Project](#initializing-the-project) - -- [Enforcing Type Checking with Pyright](#enforcing-type-checking-with-pyright) - - [Installing Pyright](#installing-pyright) - - [Configuring Pyright](#configuring-pyright) - - [Integrating Pyright with Pulumi](#integrating-pyright-with-pulumi) - - [Editor Integration](#editor-integration) - -- [Using Pythonic Input Types with TypedDict](#using-pythonic-input-types-with-typeddict) - - [Advantages of TypedDict](#advantages-of-typeddict) - - [Implementing TypedDict in Resources](#implementing-typeddict-in-resources) - - [Example: Kubernetes Deployment](#example-kubernetes-deployment) - -- [Best Practices](#best-practices) - - [Adherence to DRY Principle](#adherence-to-dry-principle) - - [Modular Code Organization](#modular-code-organization) - - [Consistent Coding Standards](#consistent-coding-standards) - -- [Code Standards and Guidelines](#code-standards-and-guidelines) - - [Naming Conventions](#naming-conventions) - - [Type Annotations](#type-annotations) - - [Error Handling](#error-handling) - -- [Further Reading](#further-reading) - -## Project Setup - -### Dependency Management with Poetry - -Ensure that `poetry` is added to your system's PATH. Refer to the [official installation guide](https://python-poetry.org/docs/#installation) for detailed instructions. - -### Initializing the Project - -1. **Initialize Poetry:** - -If not already initialized, set up Poetry in the project directory: - -```bash -poetry install -``` - -This command will create a virtual environment and install all dependencies specified in `pyproject.toml`. - -We use [Poetry](https://python-poetry.org/) for dependency management and packaging. Poetry ensures that our development environment is consistent, dependencies are properly managed, and collaboration is streamlined. - -2. **Activate the Virtual Environment:** - -```bash -poetry shell -``` - -Alternatively, you can prefix commands with `poetry run`. - -3. **Configure Pulumi to Use Poetry:** - -Ensure that `Pulumi.yaml` specifies Poetry as the toolchain: - -```yaml -name: your-pulumi-project -runtime: - name: python - options: - toolchain: poetry -``` - -4. **Install Pulumi Dependencies:** - -```bash -pulumi install -``` - -This command ensures that Pulumi recognizes and utilizes the Poetry-managed environment. - -## Enforcing Type Checking with Pyright - -Type checking enhances code reliability and maintainability. We enforce strict type checking using [Pyright](https://github.com/microsoft/pyright). - -### Installing Pyright - -Add Pyright to the development dependencies: - -```bash -poetry add --dev pyright -``` - -### Configuring Pyright - -Create a `pyrightconfig.json` in the project root to define Pyright settings: - -```json -{ - "include": ["**/*.py"], - "exclude": ["**/__pycache__/**"], - "reportMissingImports": true, - "pythonVersion": "3.8", - "typeCheckingMode": "strict" -} -``` - -- **`include`**: Files to include in type checking. -- **`exclude`**: Files or directories to exclude. -- **`reportMissingImports`**: Reports errors for unresolved imports. -- **`pythonVersion`**: Target Python version. -- **`typeCheckingMode`**: Set to `"strict"` for comprehensive checking. - -### Integrating Pyright with Pulumi - -Configure Pulumi to run Pyright before deployments: - -1. **Update `Pulumi.yaml`:** - -```yaml -name: your-pulumi-project -runtime: - name: python - options: - typechecker: pyright -``` - -2. **Run Pulumi Commands:** - -Pyright will automatically run during Pulumi operations: - -```bash -pulumi up -``` - -If type errors are detected, the deployment will halt, and errors will be displayed. - -### Editor Integration - -For real-time type checking and enhanced development experience: - -#### Visual Studio Code with Pylance - -1. **Install Pylance Extension:** - - Install the [Pylance extension](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance). - -2. **Configure Type Checking Mode:** - -Add the following to `settings.json`: - -```json -{ - "python.analysis.typeCheckingMode": "strict" -} -``` - -## Using Pythonic Input Types with TypedDict - -We leverage `TypedDict` to define resource inputs using dictionaries with type hints. This approach combines flexibility with type safety. - -### Advantages of TypedDict - -- **Conciseness**: Reduces boilerplate code. -- **Readability**: Enhances code clarity. -- **Type Safety**: Enables static type checking. -- **Flexibility**: Allows easy manipulation of configurations. - -### Implementing TypedDict in Resources - -Define custom `TypedDict` classes to specify the expected structure of resource configurations. - -#### Example - -```python -from typing import TypedDict - -class VpcArgs(TypedDict): - cidr_block: str - tags: dict - -vpc_args: VpcArgs = { - "cidr_block": "10.0.0.0/16", - "tags": { - "Name": "my-vpc", - }, -} -``` - -### Example: Kubernetes Deployment - -Implementing a Kubernetes Deployment using `TypedDict`: - -```python -from typing import TypedDict, List -from pulumi_kubernetes.apps.v1 import Deployment - -class ContainerPort(TypedDict): - containerPort: int - -class Container(TypedDict): - name: str - image: str - ports: List[ContainerPort] - -class PodSpec(TypedDict): - containers: List[Container] - -class PodTemplateSpec(TypedDict): - metadata: dict - spec: PodSpec - -class DeploymentSpec(TypedDict): - replicas: int - selector: dict - template: PodTemplateSpec - -app_labels = {"app": "nginx"} - -deployment_spec: DeploymentSpec = { - "replicas": 3, - "selector": {"matchLabels": app_labels}, - "template": { - "metadata": {"labels": app_labels}, - "spec": { - "containers": [ - { - "name": "nginx", - "image": "nginx:1.14.2", - "ports": [{"containerPort": 80}], - }, - ], - }, - }, -} - -deployment = Deployment( - "nginx-deployment", - metadata={ - "name": "nginx-deployment", - "labels": app_labels - }, - spec=deployment_spec, -) -``` - -## Best Practices - -- Use consistent key naming (e.g., `matchLabels` instead of `match_labels`) to align with Kubernetes API specifications. -- Define all necessary `TypedDict` classes to cover the nested structure of configurations. -- Leverage type annotations for variables to enhance type checking. - -### Adherence to DRY Principle - -- **Avoid Repetition**: Use variables and functions to eliminate redundant code. -- **Shared Configurations**: Centralize common configurations or parameters. - -### Modular Code Organization - -- **Packages and Modules**: Organize code into logical packages and modules. -- **Reusability**: Encapsulate functionality into reusable components. -- **Separation of Concerns**: Divide code based on functionality (e.g., networking, compute resources). - -### Consistent Coding Standards - -- **PEP 8 Compliance**: Follow Python's style guidelines. -- **Naming Conventions**: Use descriptive and consistent names for variables, functions, and resources. -- **Comments and Documentation**: Provide clear comments and docstrings where necessary. - -## Code Standards and Guidelines - -### Naming Conventions - -- __Variables and Functions__: Use `snake_case`. -- **Classes and Exceptions**: Use `PascalCase`. -- __Constants__: Use `UPPER_SNAKE_CASE`. - -### Type Annotations - -- **Mandatory Type Hints**: All functions and methods should include type hints for parameters and return types. -- **Variable Annotations**: Use type annotations for variables, especially when the type is not immediately clear. - -### Error Handling - -- **Explicit Exceptions**: Raise specific exceptions with clear messages. -- **Resource Management**: Ensure resources are properly cleaned up or rolled back on failures. -- **Logging**: Utilize logging for error reporting and debugging. - -## References - -- **Pulumi Python Documentation**: [Pulumi Python Docs](https://www.pulumi.com/docs/intro/languages/python/) -- **Poetry Documentation**: [Poetry Docs](https://python-poetry.org/docs/) -- **Pyright Documentation**: [Pyright GitHub](https://github.com/microsoft/pyright) -- **PEP 8 Style Guide**: [PEP 8](https://www.python.org/dev/peps/pep-0008/) -- **TypedDict Documentation**: [PEP 589 - TypedDict](https://www.python.org/dev/peps/pep-0589/) diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 3cf708e..0000000 --- a/docs/README.md +++ /dev/null @@ -1,90 +0,0 @@ -# Konductor Documentation - -Welcome to the Konductor documentation! This comprehensive guide covers everything you need to know about using, developing for, and contributing to the Konductor Infrastructure as Code (IaC) platform. - -## Introduction - -Konductor is a powerful Infrastructure as Code (IaC) platform built on Pulumi and Python, designed to streamline DevOps workflows and Platform Engineering practices. Whether you're a platform user deploying infrastructure, a developer contributing modules, or a maintainer managing the project, you'll find the information you need in these docs. - -## Documentation Structure - -Our documentation is organized into several main sections, each targeting specific user needs: - -### 🚀 [Getting Started](./getting_started.md) -Quick-start guide to help you begin using Konductor, including installation, basic setup, and your first deployment. - -### 📚 User Documentation -- [User Guide](./user_guide/README.md): Complete guide for platform users -- [Module Documentation](./modules/README.md): Detailed guides for individual modules -- [FAQ & Troubleshooting](./user_guide/faq_and_troubleshooting.md): Common issues and solutions - -### 💻 Developer Documentation -- [Developer Guide](./developer_guide/README.md): Guide for contributing to Konductor -- [Module Development](./developer_guide/modules/README.md): Creating and maintaining modules -- [Contribution Guidelines](./developer_guide/contribution_guidelines.md): How to contribute - -### 📖 Reference Documentation -- [Pulumi Python Standards](./reference/PULUMI_PYTHON.md): Development standards and practices -- [TypedDict Guide](./reference/TypedDict.md): Using TypedDict for configurations -- [Style Guide](./reference/style_guide.md): Documentation and code style standards - -### ⚖️ Compliance & Planning -- [Compliance Guide](./compliance/COMPLIANCE.md): Compliance standards and implementation -- [Project Roadmap](./roadmaps/ROADMAP.md): Future development plans -- [Roadmap Addendum](./roadmaps/ROADMAP_Addendum.md): Additional planning details - -## Available Modules - -Konductor includes several core modules, each with comprehensive documentation: - -### AWS Module -- [User Guide](./modules/aws/README.md) -- [Developer Guide](./developer_guide/modules/aws/README.md) -- [Implementation Roadmap](./developer_guide/modules/aws/implementation_roadmap.md) -- [EKS Setup Guide](./developer_guide/modules/aws/eks_donor_template.md) -- [OpenTelemetry Integration](./developer_guide/modules/aws/eks_opentelemetry_docs.md) - -### Cert Manager Module -- [User Guide](./modules/cert_manager/README.md) -- [Developer Guide](./developer_guide/modules/cert_manager/README.md) -- [Installation Guide](./modules/cert_manager/installation_guide.md) - -## Contributing - -We welcome contributions from the community! To get started: - -1. Read our [Contribution Guidelines](./developer_guide/contribution_guidelines.md) -2. Check our [Issue Template](./contribution_templates/issue_template.md) -3. Review our [Pull Request Template](./contribution_templates/pull_request_template.md) -4. See our [Feature Request Template](./contribution_templates/feature_request_template.md) - -## Accessibility - -This documentation adheres to web accessibility guidelines to ensure it's usable by everyone: - -- Clear heading hierarchy for easy navigation -- Alt text for all images and diagrams -- High contrast text and proper font sizing -- Keyboard-navigable interface -- Screen reader compatibility - -## Getting Help - -If you need assistance: - -1. Check the [FAQ & Troubleshooting](./user_guide/faq_and_troubleshooting.md) guide -2. Search existing [GitHub Issues](https://github.com/containercraft/konductor/issues) -3. Join our [Community Discord](https://discord.gg/Jb5jgDCksX) -4. Create a new issue using our [Issue Template](./contribution_templates/issue_template.md) - -## Documentation Updates - -This documentation is continuously improved based on user feedback and project evolution. To suggest improvements: - -1. Create an issue using our documentation issue template -2. Submit a pull request with your proposed changes -3. Join the documentation discussions in our community channels - -## License - -This documentation and the Konductor project are licensed under [LICENSE]. See the [LICENSE](../LICENSE) file for details. diff --git a/docs/call_to_action.md b/docs/call_to_action.md deleted file mode 100644 index da74cb1..0000000 --- a/docs/call_to_action.md +++ /dev/null @@ -1,248 +0,0 @@ -# Call to Action: Enhancing the Konductor Platform - -## Introduction - -This document outlines our vision for enhancing and maintaining the Konductor Infrastructure as Code (IaC) platform. It serves as both a guide for current maintainers and an invitation to potential contributors, emphasizing our commitment to quality, maintainability, and community-driven development. - -## Prime Directive - -> "Features are nice. Quality is paramount." - -Quality extends beyond code to encompass the entire developer and user experience. At Konductor, we believe that the success of open-source projects depends on the satisfaction and engagement of both community developers and users. - -## Core Principles - -### 1. Developer Experience (DX) - -- **Code Quality**: Maintain high standards through: - - Static type checking with Pyright - - Comprehensive documentation - - Automated testing - - Clear error messages - -- **Development Workflow**: - ```python - # Example of type-safe, well-documented code - from typing import TypedDict, Optional - - class ModuleConfig(TypedDict): - """Configuration for module deployment. - - Attributes: - enabled: Whether the module is enabled - version: Optional version string - namespace: Kubernetes namespace - """ - enabled: bool - version: Optional[str] - namespace: str - ``` - -### 2. User Experience (UX) - -- **Clear Documentation**: Maintain comprehensive, accessible documentation -- **Intuitive Interfaces**: Design APIs and configurations for clarity -- **Error Handling**: Provide actionable error messages -- **Progressive Disclosure**: Layer complexity appropriately - -### 3. Code Maintainability - -- **Modular Design**: - - Separate concerns clearly - - Create reusable components - - Implement consistent interfaces - -- **Type Safety**: - - Use TypedDict for configurations - - Implement strict type checking - - Provide clear type annotations - -### 4. Community Focus - -- **Open Communication**: - - Active Discord community - - Responsive issue management - - Regular updates and roadmap sharing - -- **Inclusive Development**: - - Welcome contributions of all sizes - - Provide mentorship opportunities - - Maintain helpful documentation - -## Areas for Enhancement - -### 1. Modular Design Improvements - -Current: - -```python -# Before: Mixed responsibilities -class AwsResources: - def create_vpc(self): pass - def create_database(self): pass -``` - -Target: - -```python -# After: Single responsibility -class AwsNetworking: - """Manages AWS networking resources.""" - def create_vpc(self): pass - def create_database(self): pass - -class AwsDatabase: - """Manages AWS database resources.""" - def create_database(self): pass -``` - - -### 2. Configuration Management - -- **Standardize Configurations**: - - Use TypedDict consistently - - Implement validation - - Provide clear defaults - -### 3. Documentation Improvements - -- **Structure**: Follow the new documentation organization -- **Accessibility**: Ensure documentation is accessible to all -- **Examples**: Provide clear, runnable examples - -### 4. Testing Enhancements - -- **Unit Tests**: Improve coverage -- **Integration Tests**: Add end-to-end scenarios -- **Type Checking**: Enforce strict mode - -## How to Contribute - -### 1. Code Contributions - -- Follow type safety practices -- Include comprehensive tests -- Update documentation -- Add examples where appropriate - -### 2. Documentation Contributions - -- Follow the style guide -- Include code examples -- Consider accessibility -- Update related docs - -### 3. Review Process - -- Engage in constructive review -- Test thoroughly -- Verify documentation -- Check type safety - -## Development Standards - -### Code Organization - -```python -# Example of well-organized module structure -modules/ -├── aws/ -│ ├── init.py -│ ├── types.py # TypedDict definitions -│ ├── deploy.py # Deployment logic -│ └── README.md # Module documentation -``` - -### Type Safety Requirements - -- Use TypedDict for configurations -- Enable strict type checking -- Implement proper error handling - -### Documentation Requirements - -- Clear docstrings -- Type annotations -- Usage examples -- Architecture diagrams - -## Future Vision - -### Short-term Goals - -1. **Enhanced Type Safety** - - Complete TypedDict migration - - Implement strict checking - - Add validation layers - -2. **Improved Testing** - - Increase test coverage - - Add integration tests - - Implement property testing - -### Long-term Goals - -1. **Platform Evolution** - - Multi-cloud support - - Advanced compliance - - Enhanced automation - -2. **Community Growth** - - Expand contributor base - - Improve documentation - - Regular workshops - -## Call to Action - -We invite you to join us in improving the Konductor platform: - -1. **For Developers**: - - Review our [Developer Guide](./developer_guide/README.md) - - Check our [Good First Issues](https://github.com/containercraft/konductor/issues?q=is:issue+is:open+label:"good+first+issue") - - Join our [Discord](https://discord.gg/Jb5jgDCksX) - -2. **For Users**: - - Share your use cases - - Report issues - - Suggest improvements - -3. **For Documentation**: - - Help improve clarity - - Add examples - - Fix errors - -## Getting Started - -1. **Read the Documentation**: - - [Getting Started Guide](./getting_started.md) - - [Developer Guide](./developer_guide/README.md) - - [User Guide](./user_guide/README.md) - -2. **Set Up Your Environment**: - ```bash - git clone https://github.com/containercraft/konductor.git - cd konductor - poetry install - poetry shell - ``` - -3. **Start Contributing**: - - Pick an issue - - Fork the repository - - Submit a pull request - -## Community Support - -- **Discord**: Join our [Community Discord](https://discord.gg/Jb5jgDCksX) -- **GitHub**: Open issues and discussions -- **Documentation**: Contribute to our docs - -## Conclusion - -The Konductor platform thrives on community involvement and maintains high standards for code quality, documentation, and user experience. We welcome contributions that align with our vision of creating a robust, maintainable, and user-friendly Infrastructure as Code platform. - -Remember: "Features are nice. Quality is paramount." - ---- - -**Note**: This document is actively maintained. For updates and changes, refer to our [changelog](./changelog.md). diff --git a/docs/compliance/COMPLIANCE.md b/docs/compliance/COMPLIANCE.md deleted file mode 100644 index be7ba5f..0000000 --- a/docs/compliance/COMPLIANCE.md +++ /dev/null @@ -1,498 +0,0 @@ -# Compliance Standards and Implementation Guide - -## Introduction - -This document outlines the comprehensive compliance strategy implemented in the Konductor Infrastructure as Code (IaC) platform. It details how the codebase is designed to reduce the time necessary to achieve production-ready compliance and authority to operate, while also minimizing the overhead associated with compliance maintenance and renewal audits. - -## Table of Contents - -1. [Overview](#overview) - - [Objectives](#objectives) - - [Scope](#scope) -2. [Compliance Framework](#compliance-framework) - - [Supported Standards](#supported-standards) - - [Implementation Strategy](#implementation-strategy) -3. [Development Standards](#development-standards) -4. [Implementation Guidelines](#implementation-guidelines) - - [Resource Tagging](#resource-tagging) - - [Access Control](#access-control) -5. [Validation and Testing](#validation-and-testing) -6. [Documentation Requirements](#documentation-requirements) -7. [Auditing and Reporting](#auditing-and-reporting) -8. [Security Controls](#security-controls) -9. [Compliance Automation](#compliance-automation) -10. [Maintenance and Updates](#maintenance-and-updates) -11. [Conclusion](#conclusion) - -## Overview - -### Objectives - -- Automate compliance controls within IaC workflows -- Ensure consistent policy enforcement across all environments -- Reduce manual compliance tasks and human error -- Provide clear audit trails and documentation -- Support multiple compliance frameworks (NIST, FISMA, ISO 27001) - -### Scope - -This document covers: - -- Development standards and practices -- Security controls and implementation -- Documentation requirements -- Testing and validation procedures -- Audit preparation and reporting - -## Compliance Framework - -### Supported Standards - -#### NIST Framework - -- NIST SP 800-53 -- NIST Cybersecurity Framework -- NIST Cloud Computing Standards - -#### FISMA Compliance - -- Federal Information Security Management Act requirements -- Authority to Operate (ATO) prerequisites -- Continuous monitoring requirements - -#### ISO Standards - -- ISO 27001 Information Security Management -- ISO 27017 Cloud Security -- ISO 27018 Cloud Privacy - -### Implementation Strategy - -#### Configuration Schema - -```python -from typing import TypedDict, List, Dict - -class ComplianceConfig(TypedDict): - """Compliance configuration structure. - - Attributes: - nist_controls: List of NIST control identifiers - fisma_level: FISMA impact level - iso_controls: List of ISO control identifiers - audit_logging: Audit logging configuration - encryption: Encryption requirements - """ - nist_controls: List[str] - fisma_level: str - iso_controls: List[str] - audit_logging: Dict[str, bool] - encryption: Dict[str, str] - -# Default compliance configuration -default_compliance: ComplianceConfig = { - "nist_controls": ["AC-2", "AC-3", "AU-2"], - "fisma_level": "moderate", - "iso_controls": ["A.9.2.3", "A.10.1.1"], - "audit_logging": {"enabled": True, "encrypted": True}, - "encryption": {"algorithm": "AES-256", "key_rotation": "90days"} -} -``` - -## Development Standards - -### Code Quality Requirements - -#### Type Safety - -- Use type hints for all functions and variables -- Implement TypedDict for configurations -- Enable strict type checking with Pyright - -#### Documentation - -- Comprehensive docstrings for all public APIs -- Clear inline comments for complex logic -- Up-to-date README files and user guides - -#### Testing - -- Unit tests for all functionality -- Integration tests for compliance controls -- Security testing and validation - -### Python Standards - -```python -from typing import Optional, Dict, Any -import pulumi - -def create_compliant_resource( - name: str, - config: Dict[str, Any], - compliance_tags: Optional[Dict[str, str]] = None -) -> pulumi.Resource: - """Create a resource with compliance controls. - - Args: - name: Resource name - config: Resource configuration - compliance_tags: Compliance-related tags - - Returns: - pulumi.Resource: Created resource with compliance controls - """ - if not compliance_tags: - compliance_tags = {} - - # Add required compliance tags - compliance_tags.update({ - "compliance:framework": "nist", - "compliance:control": "AC-2", - "compliance:validated": "true" - }) - - # Resource creation with compliance controls - resource = pulumi.Resource( - name, - props={**config, "tags": compliance_tags}, - opts=pulumi.ResourceOptions(protect=True) - ) - - return resource -``` - -## Implementation Guidelines - -### Resource Tagging - -#### Required Tags - -- `compliance:framework` -- `compliance:control` -- `compliance:validated` -- `compliance:owner` -- `compliance:expiration` - -#### Tag Implementation - -```python -def apply_compliance_tags( - resource: pulumi.Resource, - tags: Dict[str, str] -) -> None: - """Apply compliance tags to a resource. - - Args: - resource: The resource to tag - tags: Compliance tags to apply - """ - required_tags = { - "compliance:framework": "nist", - "compliance:control": "AC-2", - "compliance:validated": "true", - "compliance:owner": "platform-team", - "compliance:expiration": "2024-12-31" - } - - # Merge required tags with provided tags - final_tags = {**required_tags, **tags} - - # Apply tags to resource - resource.tags.apply(lambda x: {**x, **final_tags}) -``` - -### Access Control - -#### IAM Configuration - -- Implement least privilege access -- Regular access review procedures -- Role-based access control (RBAC) - -#### Authentication Requirements - -- Multi-factor authentication (MFA) -- Strong password policies -- Regular credential rotation - -## Validation and Testing - -### Compliance Testing - -```python -import pytest -from typing import Dict, Any - -def test_resource_compliance( - resource_config: Dict[str, Any], - compliance_requirements: Dict[str, Any] -) -> None: - """Test resource compliance with requirements. - - Args: - resource_config: Resource configuration to test - compliance_requirements: Compliance requirements to validate - """ - # Verify required tags - assert "compliance:framework" in resource_config["tags"] - assert "compliance:control" in resource_config["tags"] - - # Verify encryption settings - assert resource_config["encryption"]["enabled"] is True - assert resource_config["encryption"]["algorithm"] == "AES-256" - - # Verify access controls - assert resource_config["access"]["mfa_enabled"] is True - assert resource_config["access"]["minimum_permissions"] is True -``` - -### Automated Validation - -#### Pre-deployment Checks - -- Configuration validation -- Policy compliance verification -- Security control validation - -#### Continuous Monitoring - -- Real-time compliance monitoring -- Automated remediation -- Compliance reporting - -## Documentation Requirements - -### Required Documentation - -#### System Documentation - -- Architecture diagrams -- Data flow documentation -- Security controls documentation - -#### Operational Procedures - -- Incident response plans -- Change management procedures -- Backup and recovery procedures - -#### Compliance Evidence - -- Control implementation evidence -- Test results and validations -- Audit logs and reports - -#### Documentation Format - -```python -# Component Documentation Template - -## Overview -[Component description and purpose] - -## Compliance Controls -- NIST Controls: [List applicable controls] -- FISMA Requirements: [List FISMA requirements] -- ISO Controls: [List ISO controls] - -## Implementation Details -[Technical implementation details] - -## Security Controls -[Security measures and controls] - -## Testing and Validation -[Testing procedures and results] - -## Maintenance Procedures -[Routine maintenance requirements] -``` - -## Auditing and Reporting - -### Audit Logging - -```python -from typing import Dict, Any -import logging - -def setup_compliance_logging( - config: Dict[str, Any] -) -> None: - """Configure compliance audit logging. - - Args: - config: Logging configuration - """ - logging.basicConfig( - format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', - level=logging.INFO - ) - - # Add compliance-specific handlers - handler = logging.FileHandler('compliance_audit.log') - handler.setFormatter(logging.Formatter( - '%(asctime)s - %(name)s - %(levelname)s - %(message)s' - )) - logging.getLogger('compliance').addHandler(handler) -``` - -### Reporting Requirements - -#### Regular Reports - -- Monthly compliance status -- Quarterly security assessments -- Annual compliance reviews - -#### Incident Reporting - -- Security incident reports -- Compliance violation reports -- Remediation action reports - -## Security Controls - -### Encryption Requirements - -#### Data at Rest - -- AES-256 encryption -- Key management procedures -- Regular key rotation - -#### Data in Transit - -- TLS 1.2 or higher -- Certificate management -- Secure key exchange - -### Access Controls - -```python -from typing import Dict, List - -def validate_access_controls( - resource: Dict[str, Any], - required_controls: List[str] -) -> bool: - """Validate resource access controls. - - Args: - resource: Resource configuration - required_controls: Required access controls - - Returns: - bool: True if all required controls are implemented - """ - implemented_controls = resource.get("access_controls", []) - return all(control in implemented_controls for control in required_controls) -``` - -## Compliance Automation - -### Automated Controls - -#### Resource Provisioning - -- Compliant resource templates -- Automated configuration -- Validation checks - -#### Monitoring and Alerts - -- Continuous compliance monitoring -- Automated alerts -- Remediation - - workflows - -### Implementation Example - -```python -from typing import Dict, Any -import pulumi - -class ComplianceAutomation: - """Automated compliance control implementation.""" - - def __init__(self, config: Dict[str, Any]): - self.config = config - self.setup_monitoring() - self.setup_alerts() - - def setup_monitoring(self) -> None: - """Configure compliance monitoring.""" - # Implementation details - - def setup_alerts(self) -> None: - """Configure compliance alerts.""" - # Implementation details - - def validate_resource(self, resource: pulumi.Resource) -> bool: - """Validate resource compliance. - - Args: - resource: Resource to validate - - Returns: - bool: True if resource is compliant - """ - # Validation implementation - return True -``` - -## Maintenance and Updates - -### Regular Reviews - -#### Monthly Reviews - -- Control effectiveness -- Policy compliance -- Security posture - -#### Quarterly Assessments - -- Comprehensive audits -- Control updates -- Documentation reviews - -### Update Procedures - -```python -from typing import Dict, Any -import datetime - -def update_compliance_controls( - current_controls: Dict[str, Any], - new_requirements: Dict[str, Any] -) -> Dict[str, Any]: - """Update compliance controls with new requirements. - - Args: - current_controls: Current compliance controls - new_requirements: New compliance requirements - - Returns: - Dict[str, Any]: Updated compliance controls - """ - updated_controls = current_controls.copy() - - # Update controls - for control, requirement in new_requirements.items(): - updated_controls[control] = requirement - - # Add update metadata - updated_controls["last_updated"] = datetime.datetime.now().isoformat() - updated_controls["update_version"] = str( - int(current_controls.get("update_version", "0")) + 1 - ) - - return updated_controls -``` - -## Conclusion - -This compliance framework provides a comprehensive approach to maintaining security and regulatory compliance within the Konductor IaC platform. By following these guidelines and implementing the provided controls, organizations can achieve and maintain compliance while minimizing operational overhead. Regular reviews and updates of this document ensure that it remains current with evolving compliance requirements and best practices. For questions or clarification, please contact the compliance team or refer to the project documentation. diff --git a/docs/developer_guide/README.md b/docs/developer_guide/README.md index 08f4be1..a969fbe 100644 --- a/docs/developer_guide/README.md +++ b/docs/developer_guide/README.md @@ -1,6 +1,6 @@ # Konductor Developer Guide: Building a Scalable and Maintainable Pulumi Python Project -Welcome to the **Konductor Developer Guide**! This comprehensive documentation is designed to help you understand the architecture, design decisions, and best practices for contributing to the Konductor Infrastructure as Code (IaC) platform. Whether you're a junior developer new to IaC or a seasoned engineer, this guide provides the essential knowledge to develop scalable, maintainable, and efficient code in the Konductor project. +Welcome to the **Konductor Developer Guide**! This documentation is here to assist you in understanding the architecture, design decisions, and best practices for contributing to the Konductor Infrastructure as Code (IaC) platform using Pulumi with Python. Whether you're new to IaC or a seasoned developer, this guide equips you with the knowledge to develop scalable, maintainable, and efficient code within the Konductor project. --- @@ -11,7 +11,6 @@ Welcome to the **Konductor Developer Guide**! This comprehensive documentation i 3. [Getting Started](#getting-started) - [Development Environment Setup](#development-environment-setup) - [Repository Structure](#repository-structure) - - [Core Technologies](#core-technologies) 4. [Directory Structure](#directory-structure) 5. [Code Organization and Modularization](#code-organization-and-modularization) 6. [Module Development](#module-development) @@ -20,11 +19,17 @@ Welcome to the **Konductor Developer Guide**! This comprehensive documentation i - [Module Testing](#module-testing) - [Module Documentation](#module-documentation) 7. [Entry Point and Initialization](#entry-point-and-initialization) + - [Simplified Entry Point (`__main__.py`)](#simplified-entry-point-__main__py) + - [Module Initialization (`__init__.py`)](#module-initialization-__init__py) 8. [Configuration Management](#configuration-management) 9. [Dynamic Module Loading](#dynamic-module-loading) 10. [Type Safety and Static Type Checking](#type-safety-and-static-type-checking) 11. [Dependency Management with Poetry](#dependency-management-with-poetry) 12. [Coding Standards and Best Practices](#coding-standards-and-best-practices) + - [Code Style](#code-style) + - [Consistent Naming Conventions](#consistent-naming-conventions) + - [Error Handling and Exceptions](#error-handling-and-exceptions) + - [Advanced Coding Techniques](#advanced-coding-techniques) 13. [Testing and Continuous Integration](#testing-and-continuous-integration) 14. [Collaboration and Workflow](#collaboration-and-workflow) - [Contribution Workflow](#contribution-workflow) @@ -32,10 +37,10 @@ Welcome to the **Konductor Developer Guide**! This comprehensive documentation i - [Code Review Process](#code-review-process) 15. [Scaling Considerations](#scaling-considerations) 16. [Additional Considerations](#additional-considerations) + - [Security](#security) + - [Documentation Standards](#documentation-standards) + - [Logging and Monitoring](#logging-and-monitoring) 17. [Available Modules](#available-modules) - - [AWS Module](#aws-module) - - [Cert Manager Module](#cert-manager-module) - - [Other Modules](#other-modules) 18. [Conclusion](#conclusion) 19. [Additional Resources](#additional-resources) @@ -43,37 +48,46 @@ Welcome to the **Konductor Developer Guide**! This comprehensive documentation i ## Introduction -The Konductor project is a pre-written boilerplate designed to streamline platform engineering and facilitate consistent Infrastructure as Code (IaC) practices across application teams. By leveraging Pulumi with Python, the project allows teams to define infrastructure in a familiar programming language while adhering to best practices in IaC. +**Infrastructure as Code (IaC)** is the practice of managing and provisioning infrastructure through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. IaC enables you to automate infrastructure provisioning and manage it using the same version control systems and workflows as application code. -Our development approach is guided by several key principles: +**Python IaC** refers to using the Python programming language to define and manage infrastructure. Python's readability and extensive ecosystem make it a powerful choice for infrastructure automation. -- **Type Safety**: Utilize Python's type system, Pydantic models, and `TypedDict` to catch errors early. -- **Modularity**: Organize code into reusable, self-contained modules. -- **Documentation**: Maintain comprehensive documentation as a first-class citizen. -- **Testing**: Ensure reliability and maintainability through thorough testing. -- **Accessibility**: Make code and documentation accessible to developers of all skill levels. +**Pulumi** is an open-source infrastructure as code tool that allows you to define cloud resources using general-purpose programming languages like Python, TypeScript, Go, and more. Pulumi enables you to leverage familiar programming constructs, libraries, and tools to manage infrastructure across various cloud providers. + +The **Konductor** project is a pre-written boilerplate aimed at streamlining platform engineering and fostering consistent Infrastructure as Code practices across teams. Utilizing Pulumi with Python, the project enables configuration-driven infrastructure platform engineering and orchestration in a familiar programming language while ensuring adherence to best IaC practices and Pulumi's native conventions. + +Core principles guiding our development approach include: + +- **Type Safety**: Leverage Python's type system, Pydantic models, and `TypedDict` for early error detection. +- **Modularity**: Structure code into reusable, self-contained modules. +- **Documentation**: Prioritize comprehensive documentation as an integral aspect of development. +- **Testing**: Guarantee reliability and maintainability via thorough testing. +- **Accessibility**: Ensure code and documentation are accessible to developers of all levels. - **Configuration-Driven Deployment**: Enable or disable modules based on configuration for flexible deployments. -- **Lazy Loading of Modules**: Load only necessary modules to optimize performance and resource usage. +- **Lazy Loading of Modules**: Load only necessary modules to enhance performance and resource usage. +- **Secure Secrets Management**: Utilize Pulumi's Environments, Secrets, and Configuration (ESC) for secure and centralized management of secrets and configurations. -This guide provides a detailed roadmap to help you: +This guide provides a roadmap to help you: -- Understand the architectural decisions and design patterns used in Konductor. -- Learn how to organize code effectively for scalability and maintainability. +- Understand the architectural decisions and design patterns in Konductor. +- Effectively organize code for scalability and maintainability. - Implement dynamic module loading based on configuration. -- Ensure type safety and code reliability using Pydantic, `TypedDict`, and Pyright. -- Collaborate efficiently with other developers in a large codebase. +- Ensure type safety and reliability using Pydantic, `TypedDict`, and Pyright. +- Manage secrets and configurations securely using Pulumi ESC. +- Collaborate efficiently with a team in a large codebase. --- ## Project Overview -Konductor aims to: +Konductor boilerplate template IaC aims to: -- **Provide a Shared IaC Style**: Facilitate consistent infrastructure definitions across teams. -- **Enable Dynamic, Configuration-Driven Deployments**: Allow modules to be enabled or disabled based on configuration. -- **Promote Modularity and Reusability**: Encourage code reuse and separation of concerns. -- **Simplify the Entry Point**: Keep the `__main__.py` minimal, delegating complex logic to core modules. -- **Support Lazy Loading of Modules**: Load only necessary modules to optimize performance and resource usage. +- **Provide a Shared IaC Style**: Standardize infrastructure definitions across teams, promoting consistency and best practices. +- **Enable Configuration-Driven Deployments**: Allow module enablement or disablement via configuration, providing flexibility for different environments and use cases. +- **Promote Modularity and Reusability**: Encourage code reuse and separation of concerns, making the codebase easier to maintain and extend. +- **Simplify the Entry Point**: Keep `__main__.py` minimal and delegate complex logic to core modules for better readability and organization. +- **Support Lazy Loading of Modules**: Optimize performance by loading only necessary modules based on configuration. +- **Centralize Secrets and Configurations**: Use Pulumi ESC for managing secrets and configurations securely across projects and environments. --- @@ -81,56 +95,40 @@ Konductor aims to: ### Development Environment Setup +To streamline your development setup, the Konductor project leverages a comprehensive devcontainer configuration, which includes all dependencies and development tools. + #### Prerequisites -- **Python 3.8+** -- **Poetry** for dependency management -- **Pulumi CLI** -- **Git** -- **Visual Studio Code (recommended)** +- **Visual Studio Code (VSCode)**: [Download VSCode](https://code.visualstudio.com/) +- **Remote Development Extension Pack**: Install via the [VSCode Marketplace](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) +- **Docker Desktop or Docker CLI**: [Install Docker](https://docs.docker.com/get-docker/) #### Initial Setup -```bash -# Clone the repository -git clone https://github.com/containercraft/konductor.git -cd konductor +1. **Clone the Repository**: -# Install dependencies -poetry install + ```bash + git clone https://github.com/containercraft/konductor.git + cd konductor + ``` -# Activate virtual environment -poetry shell +2. **Open in VSCode**: -# Initialize Pulumi -pulumi login -``` + Open the `konductor` directory in VSCode and use the command palette (`Ctrl+Shift+P` or `Cmd+Shift+P` on Mac) to select **"Remote-Containers: Reopen in Container"** to initiate the devcontainer. -#### VS Code Configuration +3. **Start Developing**: -- **Install Recommended Extensions**: - - Python extension - - Pylance (for type checking) - - Pulumi extension -- **Configure Pylance**: - - Set `"python.analysis.typeCheckingMode": "strict"` in your `settings.json`. -- **Set Up Dev Container** (optional but recommended): - - Use the provided `.devcontainer` configuration for a consistent development environment. + All necessary tools, including Pulumi and Poetry, are pre-configured within the devcontainer, allowing for immediate development. ### Repository Structure -The repository follows a structured layout to promote clarity and ease of navigation. +The repository is organized to facilitate clarity and ease of navigation: ```plaintext -konductor/ -├── pyproject.toml -├── poetry.lock -├── Pulumi.yaml -├── Pulumi..yaml -├── README.md -├── LICENSE -├── .gitignore +../konductor/ ├── __main__.py +├── README.md +├── Pulumi.yaml ├── core/ │ ├── __init__.py │ ├── initialization.py @@ -143,22 +141,43 @@ konductor/ │ ├── __init__.py │ ├── aws/ │ │ ├── __init__.py -│ │ ├── deploy.py │ │ ├── types.py +│ │ ├── provider.py +│ │ ├── config.py +│ │ ├── deploy.py │ │ └── ... │ ├── azure/ │ │ ├── __init__.py -│ │ ├── deploy.py │ │ ├── types.py +│ │ ├── provider.py +│ │ ├── config.py +│ │ ├── deploy.py │ │ └── ... │ ├── kubernetes/ │ ├── __init__.py -│ ├── deploy.py │ ├── types.py +│ ├── deploy.py +│ ├── provider.py +│ ├── cert_manager/ +│ │ ├── __init__.py +│ │ ├── types.py +│ │ ├── config.py +│ │ └── deploy.py +│ ├── kubevirt/ +│ │ ├── __init__.py +│ │ ├── types.py +│ │ ├── config.py +│ │ └── deploy.py +│ ├── pulumi_operator/ +│ │ ├── __init__.py +│ │ ├── types.py +│ │ ├── config.py +│ │ └── deploy.py +│ ├── multus/ +│ ├── crossplane/ +│ ├── argocd/ +│ ├── flux/ │ └── ... -├── config/ -│ ├── __init__.py -│ └── base.py ├── common/ │ ├── __init__.py │ ├── utils.py @@ -171,25 +190,36 @@ konductor/ │ ├── modules/ │ └── ... ├── docs/ +│ ├── developer_guide/ +│ │ ├── README.md +│ │ └── devcontainer.md +│ ├── user_guide/ +│ │ ├── README.md +│ │ └── faq_and_troubleshooting.md +│ ├── reference/ +│ │ └── PULUMI_PYTHON.md │ └── ... +├── LICENSE +├── .gitignore +├── pyproject.toml +├── poetry.lock ``` --- ## Directory Structure -A well-organized directory structure is crucial for collaboration and scalability. The Konductor project follows a modular layout that promotes separation of concerns. +An organized directory structure is critical for collaboration and scalability, adhering to a modular layout that promotes separation of concerns. -**Explanation:** +**Outline:** -- **`konductor/`**: The top-level directory containing all source code. -- **`__main__.py`**: The simplified entry point of the Pulumi project. -- **`core/`**: Contains core functionalities such as initialization, configuration management, metadata setup, and deployment management. -- **`modules/`**: Contains individual modules for different providers (e.g., AWS, Azure, Kubernetes), each with its own deployment logic and types. -- **`config/`**: Contains base configuration classes using Pydantic models. -- **`common/`**: Holds shared utilities, custom exceptions, and common type definitions. -- **`tests/`**: Organized in parallel with the `konductor/` structure for targeted testing. -- **`docs/`**: Contains project documentation. +- **`../konductor/`**: Top-level directory containing all source code. +- **`__main__.py`**: Simplified project entry point. +- **`core/`**: Core functionalities, such as initialization, metadata setup, and deployment management. +- **`modules/`**: Provider-specific modules (e.g., AWS, Azure, Kubernetes) with deployment logic, default configuration, config validation, and types. +- **`common/`**: Shared utilities, custom exceptions, and type definitions. +- **`tests/`**: Organized structure for targeted testing. +- **`docs/`**: Project documentation. --- @@ -197,25 +227,29 @@ A well-organized directory structure is crucial for collaboration and scalabilit ### Module Structure -- **Core Modules (`core/`)**: Centralize initialization, configuration, metadata, and deployment management. -- **Modules (`modules/`)**: Each module represents a deployable unit (e.g., `aws`, `azure`, `kubernetes`) and contains its own deployment logic and types. -- **Configurations (`config/`)**: Use Pydantic models for configuration management. +- **Core Modules (`core/`)**: Centralize initialization, metadata, and deployment management. +- **Modules (`modules/`)**: Each module represents a deployable unit (e.g., `aws`, `azure`, `kubernetes/cert_manager`, `kubernetes/kubevirt`, `kubernetes/pulumi_operator`), containing its own logic and types. - **Common Utilities (`common/`)**: Include shared utilities, exceptions, and type definitions. +- **Strongly Typed**: Ensure type safety using Pydantic, `TypedDict`, and Pyright. -#### Why Modularize? +#### Benefits of Modularization -- **Separation of Concerns**: Different functionalities are isolated to enhance maintainability. -- **Lazy Loading**: Modules are loaded dynamically based on configuration, optimizing performance. -- **Team Collaboration**: Different teams can work independently on separate modules. +- **Separation of Concerns**: Enhance maintainability by isolating functionalities. +- **Lazy Loading**: Optimize performance through dynamic module loading. +- **Team Collaboration**: Facilitate independent module work by diverse teams. +- **Per Module Enablement**: Enable or disable modules based on configuration with built-in sane defaults. +- **Independent Module Configuration**: Each module maintains loosely coupled, independent configuration specs. ### File Layout Within Modules -Each module in `modules/` should contain: +Each module in `modules/` should include: - **`__init__.py`**: Initializes the module. -- **`deploy.py`**: Contains the `deploy` function that handles the deployment logic for the module. -- **`types.py`**: Defines data classes or Pydantic models for the module's configuration. -- **Additional Files**: Any other module-specific implementation files. +- **`types.py`**: Data classes or Pydantic models specifying the module's configuration. +- **`provider.py`**: Contains the provider authentication and authorization resources if applicable. +- **`config.py`**: Module-specific configuration parsing, validation, and default values, which support merging user-supplied overrides. +- **`deploy.py`**: Defines the `deploy` function containing deployment logic. +- **Additional Files**: Any other module-specific implementations. --- @@ -223,15 +257,17 @@ Each module in `modules/` should contain: ### Module Architecture -Modules follow a standard structure: +Modules follow a standardized structure: ```plaintext modules// ├── __init__.py +├── types.py # Configuration data classes or Pydantic models +├── provider.py # Cloud provider authentication and authorization resources +├── config.py # Configuration parsing, validation, and default values ├── deploy.py # Deployment logic -├── types.py # Data classes or Pydantic models for configurations ├── README.md # Module documentation -└── ... # Additional module-specific files +└── ... # Additional files ``` ### Creating New Modules @@ -241,55 +277,112 @@ To create a new module: 1. **Create Module Directory**: ```bash - mkdir modules/ + mkdir -p modules// touch modules//__init__.py + touch modules//types.py + touch modules//provider.py + touch modules//config.py + touch modules//deploy.py + ``` + +2. **Implement `provider.py`**: + + Define the provider authentication and client setup. + + ```python + # modules//provider.py + from pulumi import ResourceOptions + from pulumi_ import Provider + + def get_provider(config): + # Instantiate and return the provider + return Provider( + resource_name="", + # Pass in necessary provider configuration + opts=ResourceOptions() + ) ``` -2. **Implement `deploy.py`**: +3. **Implement `config.py`**: - - Define a `deploy` function that accepts `config` and `init_config` as parameters. + Handle configuration parsing, validation, and default values. + + ```python + # modules//config.py + from pydantic import BaseModel + + class ModuleConfig(BaseModel): + enabled: bool = True + # Additional configuration fields with defaults and validation + + def load_config(config_data) -> ModuleConfig: + return ModuleConfig(**config_data) + ``` + +4. **Implement `deploy.py`**: + + Define a `deploy` function which receives `config` and `init_config`. ```python # modules//deploy.py - from typing import Any, Dict from core.types import InitializationConfig, ModuleDeploymentResult + from .config import load_config + from .provider import get_provider + from pulumi import log + + def deploy(config_data: dict, init_config: InitializationConfig) -> ModuleDeploymentResult: + log.info(f"Deploying module ") - def deploy(config: Dict[str, Any], init_config: InitializationConfig) -> ModuleDeploymentResult: - # Deployment logic here - pass + # Load and validate module configuration + module_config = load_config(config_data) + + if not module_config.enabled: + log.info(f"Module is disabled in configuration.") + return ModuleDeploymentResult(success=True, message="Module disabled") + + # Get provider instance + provider = get_provider(module_config) + + # Deployment logic here, using provider and module_config + # ... + + return ModuleDeploymentResult(success=True, message="Deployment successful") ``` -3. **Define Module Configuration Types**: +5. **Define Module Configuration Types**: - - Use Pydantic models or `TypedDict` in `types.py` for configuration. + Use Pydantic models in `types.py` for configuration specifications. ```python # modules//types.py - from typing import TypedDict + from pydantic import BaseModel - class ModuleConfig(TypedDict): - enabled: bool - version: str + class ModuleConfig(BaseModel): + enabled: bool = True # Additional configuration fields ``` -4. **Write Module Documentation**: +6. **Write Documentation**: - - Include a `README.md` with usage instructions, configuration details, and examples. + Include a `README.md` outlining usage instructions and configuration details. ### Module Testing -- **Unit Tests**: Write unit tests for your module's functionality in `tests/modules//`. -- **Integration Tests**: Include integration tests to verify module interactions if applicable. +- **Unit Tests**: Located in `tests/modules//`. +- **Integration Tests**: If applicable, cover module interactions. +- **Testing Best Practices**: + - Ensure tests are isolated and repeatable. + - Use mock objects where appropriate. + - Cover edge cases and error conditions. ### Module Documentation -Each module must include: +Each module must feature: -- **README.md**: Usage instructions, configuration options, and examples. -- **Configuration Documentation**: Detailed explanation of configuration parameters. -- **Example Configurations**: Provide sample configurations for users. -- **Troubleshooting Guide**: Common issues and solutions. +- **README.md**: Details on usage, configuration, and examples. +- **Configuration Documentation**: Explanation of parameters. +- **Complete Explicit Example Configuration**: Provide samples for easy understanding. +- **Troubleshooting Guide**: Address common issues and solutions. --- @@ -297,7 +390,7 @@ Each module must include: ### Simplified Entry Point (`__main__.py`) -The `__main__.py` serves as the minimal entry point, delegating complex logic to core modules. +The `__main__.py` serves as a minimal entry point, delegating complex logic to core modules. #### Structure: @@ -320,7 +413,7 @@ def main() -> None: # Set up global metadata setup_global_metadata(init_config) - # Get the list of enabled modules from the configuration + # Determine enabled modules from configuration modules_to_deploy = get_enabled_modules(init_config.config) log.info(f"Deploying modules: {', '.join(modules_to_deploy)}") @@ -340,63 +433,106 @@ if __name__ == "__main__": #### Best Practices: -- **Keep Entry Point Minimal**: Only include essential orchestration steps. -- **Delegate Logic**: Move complex operations to core modules. -- **Dynamic Module Loading**: Ensure only necessary modules are loaded based on configuration. +- **Minimalistic Entry Point**: Focus on key orchestration tasks. +- **Delegated Logic**: Shift complex logic to core modules. +- **Dynamic Module Loading**: Load only the necessary modules. ### Module Initialization (`__init__.py`) -- **Avoid Heavy Logic**: Keep initialization lightweight. -- **Expose Public API**: Define accessible elements when the module is imported. +- **Lightweight**: Avoid heavy logic in initialization files. +- **Public API**: Define accessible elements upon importing. --- ## Configuration Management -### Using Pydantic for Configuration +### Using Pulumi ESC for Configuration and Secrets Management -Pydantic models provide type safety and validation for configuration data. +Pulumi Environments, Secrets, and Configuration (ESC) provides a centralized and secure way to manage configurations and secrets across your Pulumi projects. It allows you to compose collections of configuration and secrets called environments, which can be consumed by various infrastructure and application services. -#### Example: +#### Setting Up Pulumi ESC -```python -# config/base.py -from pydantic import BaseSettings -from typing import List +1. **Install the ESC CLI**: + + ```bash + curl -fsSL https://get.pulumi.com/install.sh | sh + ``` -class BaseConfig(BaseSettings): - environment: str = 'development' - debug: bool = False - modules: List[str] = [] +2. **Log in to Pulumi Cloud**: - class Config: - env_file = '.env' -``` + ```bash + pulumi login + ``` + +3. **Create an ESC Environment**: + + ```bash + esc env init konductor/dev + ``` + +4. **Define Configuration Values**: + + Create a `values.yaml` file with your configuration: + + ```yaml + # values.yaml + values: + pulumiConfig: + aws:region: us-west-2 + aws:profile: default + secrets: + db_password: your-database-password + ``` + + Import the values into your environment: -#### Usage: + ```bash + esc env set-values konductor/dev -f values.yaml + ``` + +5. **Import the Environment into Your Pulumi Project**: + + Update your `Pulumi..yaml`: + + ```yaml + environment: + - konductor/dev + ``` + +#### Accessing Configuration in Code ```python # core/initialization.py import pulumi -from config.base import BaseConfig from core.types import InitializationConfig +from pulumi import log def initialize_pulumi() -> InitializationConfig: # Load Pulumi configuration pulumi_config = pulumi.Config() - # Create BaseConfig instance - base_config = BaseConfig() + # Access configuration values + aws_region = pulumi_config.require("aws:region") + db_password = pulumi_config.require_secret("db_password") + log.info(f"Pulumi initialization complete. AWS Region: {aws_region}") # Return an InitializationConfig instance - return InitializationConfig(config=pulumi_config, base_config=base_config) + return InitializationConfig(config=pulumi_config) ``` +#### Pulumi ESC Benefits + +- **Centralized Management**: Manage secrets and configurations across multiple projects and environments. +- **Security**: Secrets are encrypted in transit and at rest. +- **Versioning**: ESC supports rich versioning features for auditability and controlled configuration change rollout. +- **Projects and Environment Tags**: Organize environments using Projects and tag them for better management. + #### Best Practices: -- **Environment Variables**: Use environment variables for sensitive information. -- **Validation**: Leverage Pydantic's validation to ensure configuration correctness. -- **Default Values**: Provide sensible defaults for configuration parameters. +- **Use ESC for Secrets Management**: Avoid storing secrets in code, local configuration files, or environment variables. +- **Leverage Projects and Tags**: Organize environments logically using Projects and Environment Tags. +- **Version Control**: Utilize ESC's versioning to manage changes to environments safely. +- **Avoid Hardcoding Configuration**: Use configuration files and Pulumi ESC to manage values dynamically. --- @@ -404,15 +540,17 @@ def initialize_pulumi() -> InitializationConfig: ### Configuration-Driven Deployment -Modules are enabled or disabled based on configuration, and only enabled modules are loaded and deployed. +Modules are loaded based on configuration, facilitating dynamic and efficient deployments. #### Example: ```python # core/config.py from typing import List +from pulumi import log +import pulumi -def get_enabled_modules(config: dict) -> List[str]: +def get_enabled_modules(config: pulumi.Config) -> List[str]: default_modules = { "aws": False, "azure": False, @@ -420,13 +558,16 @@ def get_enabled_modules(config: dict) -> List[str]: } # Load user configuration for modules - modules_config = config.get("modules") or {} + modules_config = config.get_object("modules") or {} # Merge configurations merged_modules = {**default_modules, **modules_config} + enabled_modules = [module for module, enabled in merged_modules.items() if enabled] + log.info(f"Enabled modules: {enabled_modules}") + # Return enabled modules - return [module for module, enabled in merged_modules.items() if enabled] + return enabled_modules ``` ### Deployment Manager @@ -468,11 +609,13 @@ class DeploymentManager: module_config = self.init_config.config.get_object(module_name) or {} # Call the deploy function - result = deploy_func(config=module_config, init_config=self.init_config) + result = deploy_func(config_data=module_config, init_config=self.init_config) # Store the deployment result self.deployed_modules[module_name] = result + log.info(f"Successfully deployed module {module_name}.") + except ImportError as e: log.error(f"Module {module_name} could not be imported: {str(e)}") raise @@ -485,25 +628,38 @@ class DeploymentManager: ## Type Safety and Static Type Checking -### Using TypedDict for Configurations +### Using TypedDict and Pydantic for Configurations -`TypedDict` helps define type-safe dictionaries for configurations. +`TypedDict` and Pydantic models allow for defined type-safe configurations, enabling type checking and flexibility. -#### Example: +#### Example with TypedDict: ```python # common/types.py from typing import TypedDict -class ModuleConfig(TypedDict): +class ModuleConfig(TypedDict, total=False): enabled: bool version: str config: dict ``` +#### Example with Pydantic: + +```python +# modules/aws/types.py +from pydantic import BaseModel + +class AWSModuleConfig(BaseModel): + enabled: bool = True + region: str + profile: str = "default" + # Additional configuration fields +``` + ### Enforcing Type Checking with Pyright -Pyright is used for static type checking to ensure code reliability. +Integrate Pyright for static type checking, ensuring code reliability and early detection of type errors. #### Installation: @@ -513,29 +669,30 @@ poetry add --dev pyright #### Configuration: -Create a `pyrightconfig.json` in the project root: +Create `pyrightconfig.json`: ```json { "include": ["konductor/**/*.py"], "exclude": ["**/__pycache__/**"], "reportMissingImports": true, - "pythonVersion": "3.8", + "pythonVersion": "3.10", "typeCheckingMode": "strict" } ``` #### Best Practices: -- **Mandatory Type Hints**: All functions and methods must have type hints. -- **Consistent Usage**: Apply `TypedDict` and Pydantic models across the project. -- **Editor Integration**: Configure your editor (e.g., VS Code) for real-time type checking. +- **Mandatory Type Hints**: Ensure all functions and methods are type hinted. +- **Consistent Usage**: Consistently apply `TypedDict` and Pydantic models. +- **Editor Integration**: Ensure editor support for real-time type checking. +- **Validate Configuration**: Use Pydantic for robust configuration validation. --- ## Dependency Management with Poetry -Poetry manages dependencies and virtual environments. +Poetry simplifies management of dependencies and virtual environments, ensuring compatibility with Pulumi's ecosystem. ### Initializing the Project @@ -559,7 +716,7 @@ name: konductor runtime: name: python options: - virtualenv: null + virtualenv: poetry ``` ### Scripts and Entry Points @@ -577,21 +734,61 @@ konductor = 'konductor.__main__:main' ### Code Style -- **PEP 8 Compliance**: Follow standard Python style guidelines. -- **Type Hinting**: Use type annotations consistently. -- **Docstrings**: Document modules, classes, and functions using docstrings. +- **PEP 8 Compliance**: Adhere to Python style guidelines for consistency. +- **Type Hinting**: Use type annotations consistently across the codebase. +- **Docstrings**: Document modules, classes, and functions using informative docstrings. ### Consistent Naming Conventions - **Modules and Files**: Use `snake_case`. -- **Classes**: Use `PascalCase`. -- **Variables and Functions**: Use `snake_case`. +- **Classes**: Utilize `PascalCase`. +- **Variables and Functions**: Apply `snake_case`. ### Error Handling and Exceptions -- **Custom Exceptions**: Define in `common/exceptions.py`. -- **Consistent Error Handling**: Use `try-except` blocks appropriately. -- **Logging**: Use Python's `logging` module with appropriate log levels. +- **Custom Exceptions**: Define them in `common/exceptions.py`. +- **Consistent Error Handling**: Apply `try-except` blocks appropriately, with Pulumi's `log` for error logging. +- **Logging**: Ensure consistent logging using Pulumi's `pulumi.log` module imported as `from pulumi import log`. + +### Advanced Coding Techniques + +#### Clean and Readable Code Practices + +Adopt coding patterns that enhance code readability and maintainability. + +- **Using `*args` and `**kwargs`**: For flexible function arguments in resource definitions. +- **List Comprehensions**: Simplify the creation of multiple similar resources. +- **Utilizing `zip()`**: For parallel configuration of resources when appropriate and scalable. +- **Merging Dictionaries**: Combine default and environment-specific configurations efficiently. +- **Chaining Comparisons**: Use chained comparison operators in conditional statements for clarity. +- **Ternary Operators**: Simplify conditional assignments. +- **Favor Pulumi Stack Configuration**: Leverage stack configuration and Pulumi ESC environments to drive resource creation and reduce hardcoding. + +#### Example: Creating Multiple Resources with List Comprehensions + +```python +import pulumi_aws as aws +from pulumi import log +import pulumi + +# Load bucket configurations from Pulumi configuration +config = pulumi.Config() +bucket_configs = config.require_object("buckets") + +buckets = [] +for config in bucket_configs: + try: + bucket = aws.s3.Bucket( + resource_name=f"{config['name']}-bucket", + bucket=f"mycompany-{config['name']}-bucket", + versioning={ + "enabled": config.get("versioning", False) + }, + ) + buckets.append(bucket) + except Exception as e: + log.error(f"Failed to create bucket {config['name']}: {str(e)}") +``` --- @@ -600,14 +797,19 @@ konductor = 'konductor.__main__:main' ### Testing - **Structure**: Mirror the application structure within `tests/`. -- **Frameworks**: Use `pytest` for testing. -- **Coverage**: Aim for meaningful tests with high coverage. +- **Frameworks**: Utilize `pytest` for comprehensive testing. +- **Coverage**: Target meaningful tests with high coverage. +- **Testing Best Practices**: + - Write unit tests for individual components. + - Write integration tests for module interactions. + - Use mock providers to avoid real infrastructure changes during tests. ### Continuous Integration -- **Automated Testing**: Integrate tests into CI/CD pipelines. +- **Automated Testing**: Integrate testing into CI/CD workflows using tools like GitHub Actions, Jenkins, or CircleCI. - **Code Quality Checks**: Use `flake8`, `black`, and Pyright for linting and formatting. -- **Static Type Checking**: Enforce with Pyright. +- **Static Type Checking**: Enforce with Pyright for type safety. +- **Security Scanning**: Implement tools for dependency vulnerability scanning. --- @@ -616,10 +818,10 @@ konductor = 'konductor.__main__:main' ### Contribution Workflow 1. **Fork the Repository**: Create a personal fork of the Konductor repository. -2. **Create a Feature Branch**: Use a descriptive name for your branch. +2. **Create a Feature Branch**: ```bash - git checkout -b feature/your-feature-name + git checkout -b feature/your-username/your-feature-name ``` 3. **Make Changes**: Develop your feature or fix. @@ -630,30 +832,32 @@ konductor = 'konductor.__main__:main' pyright ``` -5. **Commit Changes**: Write clear and descriptive commit messages. +5. **Commit Changes**: Craft clear and descriptive commit messages. 6. **Push to Your Fork**: ```bash - git push origin feature/your-feature-name + git push origin feature/your-username/your-feature-name ``` 7. **Submit a Pull Request**: Open a PR against the `main` branch of the Konductor repository. ### Pull Request Guidelines -- **Follow the PR Template**: Include necessary details as per the [Pull Request Template](../contribution_templates/pull_request_template.md). +- **PR Template**: Follow the [Pull Request Template](../contribution_templates/pull_request_template.md). - **Link Issues**: Reference any related issues. -- **Include Tests**: Ensure your changes are covered by tests. -- **Pass All Checks**: Your PR should pass CI checks before review. +- **Include Tests**: Ensure changes are adequately covered by tests. +- **Pass All Checks**: CI checks must succeed before review. +- **Descriptive Title and Description**: Provide clear context for reviewers. ### Code Review Process -All contributions undergo review for: +All contributions undergo reviews examining: - **Code Quality**: Adherence to coding standards and best practices. -- **Documentation**: Completeness and clarity of documentation. +- **Documentation**: Clarity and completeness of documentation. - **Test Coverage**: Adequate tests for new functionality. -- **Type Safety**: Compliance with type checking. +- **Type Safety**: Compliance with type checking standards. +- **Security Considerations**: Ensure no security vulnerabilities are introduced. --- @@ -661,13 +865,15 @@ All contributions undergo review for: ### Performance Optimization -- **Lazy Loading**: Load modules dynamically to optimize resource usage. +- **Lazy Loading**: Optimize performance with dynamic module loading. - **Caching**: Implement caching strategies if applicable. +- **Resource Management**: Ensure efficient use of resources to prevent over-provisioning. ### Scalability -- **Horizontal Scaling**: Design for concurrent deployments if necessary. -- **Load Balancing**: Implement strategies to distribute workloads in the application layer. +- **Horizontal Scaling**: Design for concurrent deployments and scaling out. +- **Load Balancing**: Implement effective workload distribution strategies. +- **Stateless Design**: Favor stateless components to enhance scalability. --- @@ -675,18 +881,45 @@ All contributions undergo review for: ### Security -- **Secure Coding Practices**: Follow best practices to prevent vulnerabilities. -- **Dependency Management**: Regularly update dependencies to patch security issues. +The Konductor project prioritizes security and compliance, ensuring that infrastructure is provisioned and managed according to industry standards. + +#### Secure Coding Practices + +- **Least Privilege**: Implement least privilege access in resource configurations. +- **Access Control**: Utilize role-based access control (RBAC) and enforce strong authentication mechanisms. +- **Data Protection**: Ensure encryption of data at rest and in transit using industry-standard algorithms. +- **Secrets Management**: Manage sensitive information using Pulumi's Environments, Secrets, and Configuration (ESC) features. + +#### Compliance Frameworks + +Konductor supports compliance with various industry standards: + +- **NIST Framework** +- **FISMA Compliance** +- **ISO 27001, 27017, 27018** +- **HIPAA Compliance** (if applicable) +- **PCI DSS Compliance** (if applicable) + +For detailed guidelines on implementing compliance controls within your modules, refer to the [SecOps Compliance Standards and Implementation Guide](../docs/reference/SECOPS_COMPLIANCE.md). + +#### Auditing and Reporting + +- **Audit Logging**: Implement comprehensive logging for all operations. +- **Monitoring**: Utilize monitoring tools to track security events and resource health. +- **Incident Response**: Define procedures for responding to security incidents. ### Documentation Standards -- **Structure**: Organize documentation logically. -- **Accessibility**: Ensure documentation is accessible and user-friendly. +- **Logical Structure**: Ensure well-organized documentation. +- **Clarity**: Maintain clear and accessible documentation for users. +- **Updates**: Keep documentation up to date with code changes. +- **Usage Examples**: Provide practical examples to aid understanding. ### Logging and Monitoring -- **Centralized Logging**: Implement for easier monitoring and debugging. -- **Monitoring Tools**: Use tools to track application performance and health. +- **Centralized Logging**: Implement centralized logging mechanisms for easier monitoring and debugging. +- **Observability**: Integrate observability tools to gain insights into infrastructure performance and issues. +- **Alerting**: Set up alerts for critical events and thresholds. --- @@ -697,13 +930,61 @@ All contributions undergo review for: - **Developer Guide**: [modules/aws/README.md](./modules/aws/README.md) - **Implementation Roadmap**: [modules/aws/ROADMAP.md](./modules/aws/ROADMAP.md) -### Cert Manager Module +#### AWS Module Overview + +The AWS module provides resources and configurations for deploying infrastructure on Amazon Web Services. It includes functionalities such as: + +- VPC creation and management +- EC2 instances +- S3 buckets +- IAM roles and policies +- Elastic Load Balancers + +### Kubernetes Modules + +#### Cert Manager Module + +- **Documentation**: [modules/kubernetes/cert_manager/README.md](./modules/kubernetes/cert_manager/README.md) + +Provides automated TLS certificate management in Kubernetes clusters using cert-manager. + +#### KubeVirt Module + +- **Documentation**: [modules/kubernetes/kubevirt/README.md](./modules/kubernetes/kubevirt/README.md) + +Enables running virtual machine workloads natively on Kubernetes clusters. -- **Developer Guide**: [modules/cert_manager/README.md](./modules/cert_manager/README.md) +#### Crossplane Module + +- **Documentation**: [modules/kubernetes/crossplane/README.md](./modules/kubernetes/crossplane/README.md) + +Facilitates the deployment and management of cloud infrastructure using Kubernetes-native APIs. + +#### Pulumi Operator Module + +- **Documentation**: [modules/kubernetes/pulumi_operator/README.md](./modules/kubernetes/pulumi_operator/README.md) + +Integrates Pulumi deployments with Kubernetes, allowing infrastructure to be managed via Kubernetes resources. + +#### Additional Kubernetes Modules + +Explore other Kubernetes-related modules in the [modules/kubernetes/](./modules/kubernetes/) directory, including: + +- **Multus**: Enables multiple network interfaces in Kubernetes pods. +- **ArgoCD**: Provides continuous delivery tools for Kubernetes. +- **Flux**: Offers GitOps continuous delivery solutions. ### Other Modules -See our [Modules Directory](./modules/README.md) for a complete list. +Explore our [Modules Directory](./modules/README.md) for a complete list of available modules, including those for Azure, GCP, and other cloud providers. + +--- + +## Conclusion + +The Konductor project provides a robust and flexible framework for managing infrastructure as code using Pulumi and Python. By adhering to the best practices and guidelines outlined in this developer guide, you contribute to a maintainable, scalable, and efficient codebase that empowers teams across the organization. + +We encourage you to explore the available modules, contribute new features, and collaborate with the community to continuously improve the Konductor platform. --- @@ -712,26 +993,37 @@ See our [Modules Directory](./modules/README.md) for a complete list. ### Reference Documentation - **Pulumi Python Standards**: [../reference/PULUMI_PYTHON.md](../reference/PULUMI_PYTHON.md) +- **Pulumi ESC Documentation**: [Pulumi ESC](https://www.pulumi.com/docs/intro/concepts/config/#pulumi-esc) - **TypedDict Guide**: [../reference/TypedDict.md](../reference/TypedDict.md) - **Style Guide**: [../reference/style_guide.md](../reference/style_guide.md) +- **SecOps Compliance Standards**: [../reference/SECOPS_COMPLIANCE.md](../reference/SECOPS_COMPLIANCE.md) ### Community Resources - **Discord Community**: [https://discord.gg/Jb5jgDCksX](https://discord.gg/Jb5jgDCksX) -- **GitHub Discussions**: [https://github.com/containercraft/konductor/discussions](https://github.com/containercraft/konductor/discussions) +- **GitHub Discussions**: [GitHub Discussions](https://github.com/containercraft/konductor/discussions) ### Getting Help -- **Join our Discord**: [https://discord.gg/Jb5jgDCksX](https://discord.gg/Jb5jgDCksX) -- **Open an Issue**: [https://github.com/containercraft/konductor/issues](https://github.com/containercraft/konductor/issues) -- **Check our FAQ**: [../user_guide/faq_and_troubleshooting.md](../user_guide/faq_and_troubleshooting.md) +- **Join our Discord**: [Discord](https://discord.gg/Jb5jgDCksX) +- **Open an Issue**: [GitHub Issues](https://github.com/containercraft/konductor/issues) +- **Check our FAQ**: [FAQ](../user_guide/faq_and_troubleshooting.md) ### External Resources -- **Pulumi Python Documentation**: [https://www.pulumi.com/docs/intro/languages/python/](https://www.pulumi.com/docs/intro/languages/python/) -- **Pyright Documentation**: [https://github.com/microsoft/pyright](https://github.com/microsoft/pyright) -- **Poetry Documentation**: [https://python-poetry.org/docs/](https://python-poetry.org/docs/) -- **PEP 8 Style Guide**: [https://www.python.org/dev/peps/pep-0008/](https://www.python.org/dev/peps/pep-0008/) -- **Python Logging Documentation**: [https://docs.python.org/3/library/logging.html](https://docs.python.org/3/library/logging.html) -- **TypedDict Documentation**: [https://www.python.org/dev/peps/pep-0589/](https://www.python.org/dev/peps/pep-0589/) -- **Pydantic Documentation**: [https://pydantic-docs.helpmanual.io/](https://pydantic-docs.helpmanual.io/) +- **Pulumi ESC Documentation**: [Pulumi ESC](https://www.pulumi.com/docs/intro/concepts/config/#pulumi-esc) +- **Pulumi CrossGuard Documentation**: [Pulumi CrossGuard](https://www.pulumi.com/docs/guides/crossguard/) +- **Pulumi Kubernetes Operator**: [Pulumi Kubernetes Operator](https://www.pulumi.com/docs/guides/continuous-delivery/pulumi-kubernetes-operator/) +- **Pulumi Kubernetes Provider**: [Pulumi Kubernetes Provider](https://www.pulumi.com/docs/intro/cloud-providers/kubernetes/) +- **Pulumi Python Documentation**: [Pulumi Documentation](https://www.pulumi.com/docs/intro/languages/python/) +- **Kubernetes Crossplane Documentation**: [Crossplane](https://crossplane.io/docs/) +- **Pyright Documentation**: [Pyright](https://github.com/microsoft/pyright) +- **Poetry Documentation**: [Poetry](https://python-poetry.org/docs/) +- **PEP 8 Style Guide**: [PEP 8](https://www.python.org/dev/peps/pep-0008/) +- **Python Logging Documentation**: [Logging](https://docs.python.org/3/library/logging.html) +- **TypedDict Documentation**: [TypedDict](https://www.python.org/dev/peps/pep-0589/) +- **Pydantic Documentation**: [Pydantic](https://pydantic-docs.helpmanual.io/) + +--- + +**Note**: This guide is intended to be comprehensive and informative for developers and DevOps personnel of all experience levels, including those unfamiliar with Infrastructure as Code and Python-based IaC. If you have any questions or need further assistance, please reach out through our community channels. diff --git a/docs/developer_guide/ROADMAP.md b/docs/developer_guide/ROADMAP.md new file mode 100644 index 0000000..c5474bd --- /dev/null +++ b/docs/developer_guide/ROADMAP.md @@ -0,0 +1,298 @@ +# Konductor Platform Engineering Roadmap + +## Introduction + +The **Konductor Platform Engineering Roadmap** outlines our strategic vision and implementation plan to develop a cloud-agnostic, compliance-ready infrastructure platform. This initiative is designed to accelerate application development and deployment while maintaining the highest standards of security, compliance, and operational excellence. + +This roadmap serves as a comprehensive guide for all stakeholders, including developers, DevOps engineers, project managers, executive leadership, and end-users. It aims to build consensus and foster collaboration by detailing the benefits, strategies, and implementation phases of the Konductor platform. + +--- + +## Table of Contents + +1. [Executive Summary](#executive-summary) +2. [Vision and Goals](#vision-and-goals) +3. [Strategic Pillars](#strategic-pillars) +4. [Desired Impact](#desired-impact) +5. [Technical Architecture](#technical-architecture) +6. [Implementation Phases](#implementation-phases) +7. [Development Standards](#development-standards) +8. [Key Components](#key-components) +9. [Success Metrics](#success-metrics) +10. [Risk Management](#risk-management) +11. [Conclusion](#conclusion) +12. [Appendices](#appendices) + - [A. Technical Specifications](#a-technical-specifications) + - [B. Compliance Requirements](#b-compliance-requirements) + - [C. Reference Architectures](#c-reference-architectures) + - [D. Development Tools](#d-development-tools) +13. [Related Documentation](#related-documentation) + +--- + +## Executive Summary + +In today's rapidly evolving technological landscape, organizations face the challenge of deploying applications quickly while ensuring security, compliance, and scalability. The Konductor Platform Engineering initiative addresses these challenges by providing a unified, cloud-agnostic infrastructure platform. + +By leveraging modern Infrastructure as Code (IaC) practices, we aim to reduce operational complexities, enhance developer productivity, and maintain rigorous compliance standards. This roadmap details how we will achieve these objectives through strategic planning, technical excellence, and cross-functional collaboration. + +--- + +## Vision and Goals + +### Vision + +To establish a robust, scalable, and secure infrastructure platform that is cloud-agnostic and compliance-ready, enabling rapid and efficient application development and deployment across multiple cloud environments. + +### Primary Objectives + +- **Cloud-Agnostic Infrastructure**: Develop a platform that seamlessly supports multiple cloud providers (AWS, Azure, GCP). +- **Automated Compliance and Security**: Integrate compliance controls and security best practices into the infrastructure by default. +- **Enhanced Developer Experience**: Provide tools and processes that simplify development workflows and reduce time-to-production. +- **Operational Excellence**: Achieve high availability, reliability, and scalability through automation and standardized practices. + +--- + +## Strategic Pillars + +### 1. Cloud-Agnostic Architecture + +Building a platform that abstracts the complexities of different cloud providers allows us to: + +- **Increase Flexibility**: Easily switch or integrate multiple cloud services based on organizational needs. +- **Avoid Vendor Lock-In**: Maintain independence from any single cloud provider's ecosystem. +- **Optimize Costs**: Leverage the most cost-effective services across providers. + +### 2. Compliance Automation + +Embedding compliance into our infrastructure ensures that: + +- **Regulatory Requirements Are Met**: Adherence to standards like NIST, FISMA, and ISO 27001. +- **Security Risks Are Minimized**: Proactive enforcement of security policies reduces vulnerabilities. +- **Audit Processes Are Streamlined**: Automated reporting simplifies compliance audits. + +### 3. Developer Experience + +Enhancing the developer experience leads to: + +- **Increased Productivity**: Developers spend less time on infrastructure concerns and more on innovation. +- **Faster Time-to-Market**: Streamlined processes accelerate deployment cycles. +- **Improved Quality**: Standardized environments reduce errors and inconsistencies. + +--- + +## Desired Impact + +The successful implementation of the Konductor platform is expected to have the following impacts: + +- **Strategic Advantage**: Position the organization as a leader in technological innovation and agility. +- **Cost Efficiency**: Optimize resource utilization and reduce operational expenses through automation. +- **Risk Mitigation**: Strengthen security posture and compliance adherence, reducing the likelihood of breaches and penalties. +- **Employee Satisfaction**: Empower teams with tools and processes that make their work more fulfilling and less cumbersome. +- **Customer Trust**: Deliver reliable and secure services that enhance customer confidence and satisfaction. + +By addressing these areas, we aim to create a sustainable, competitive advantage that aligns with the organization's long-term strategic goals. + +--- + +## Technical Architecture + +### Overview + +The Konductor platform leverages Pulumi and Python to implement Infrastructure as Code (IaC) practices, emphasizing type safety, modularity, and compliance integration. + +### Core Components + +1. **Configuration Management** + + - Utilize Python's `TypedDict` for defining configuration schemas. + - Enforce strict type checking to prevent errors. + - Support layered configurations for flexibility. + +2. **Module System** + + - Adopt a modular architecture for scalability. + - Define standard interfaces to ensure compatibility. + - Enable independent development and testing of modules. + +3. **Compliance Framework** + + - Embed compliance controls directly into the infrastructure code. + - Use policy-as-code to automate compliance enforcement. + - Provide mechanisms for generating compliance reports and audit logs. + +4. **Metadata Propagation** + + - Implement a metadata propagation mechanism for cross-resource tracability and auditing. + - Enable dynamic compliance configuration updates based on metadata records. + - Automatically report compliance status to audit and monitoring systems. + +--- + +## Implementation Phases + +### Approach to Timeline Estimation + +Recognizing the complexities of engineering projects and the impact of non-development activities (e.g., meetings, holidays), we have conservatively estimated timelines to accommodate potential delays and ensure realistic expectations. + +### Phase 1: Foundations + +- **Objectives**: + - Establish the Pulumi-based infrastructure codebase. + - Implement core configuration management. + - Set up CI/CD pipelines for automation. + +### Phase 2: Core Infrastructure + +- **Objectives**: + - Deploy foundational AWS infrastructure components. + - Develop reusable networking modules. + - Integrate monitoring and logging systems. + +### Phase 3: Compliance Integration + +- **Objectives**: + - Integrate compliance controls based on regulatory standards. + - Automate compliance validation within deployment processes. + - Enhance security measures across all resources. + +### Phase 4: Multi-Cloud Expansion + +- **Objectives**: + - Extend platform support to Azure and GCP. + - Finalize the provider abstraction layer. + - Enable cross-cloud networking capabilities. + +--- + +## Development Standards + +### Code Organization + +- **Modularity**: Organize code into logically separated modules. +- **Type Safety**: Enforce type annotations and use `TypedDict`. +- **Consistency**: Follow naming conventions and code formatting standards. + +### Testing Requirements + +- **Unit Testing**: Mandatory for all functions and classes. +- **Integration Testing**: Validate interactions between modules. +- **Compliance Testing**: Ensure compliance controls are effective. + +### Documentation Standards + +- **Clarity**: Write clear and concise documentation accessible to all stakeholders. +- **Comprehensiveness**: Include detailed explanations, examples, and guidelines. +- **Consistency**: Adhere to the [Documentation Style Guide](./developer_guide/documentation.md). + +--- + +## Key Components + +### 1. Account Structure + +- **Multi-Account Strategy**: Isolate environments and applications for security and manageability. +- **Access Control**: Implement role-based access using IAM roles and policies. +- **Resource Organization**: Utilize tagging and resource groups for effective management. + +### 2. Infrastructure Components + +- **Networking**: Establish secure and scalable network architectures. +- **Compute Resources**: Provision and manage servers, containers, and serverless functions. +- **Storage Solutions**: Implement reliable and efficient storage services. +- **Security Controls**: Apply best practices for encryption, identity management, and threat detection. + +### 3. Compliance Framework + +- **Policy Definition**: Codify compliance requirements for automated enforcement. +- **Control Implementation**: Map infrastructure components to compliance controls. +- **Audit Readiness**: Maintain comprehensive logs and reports for auditing purposes. +- **Utilize Policy-as-Code**: Implement compliance rules as code for consistency and automation. +- **Adopt vendor native compliance tools**: Utilize cloud provider's compliance tools to automate compliance. + +--- + +## Success Metrics + +### Technical Metrics + +- **Deployment Reliability**: High percentage of successful deployments. +- **Infrastructure Consistency**: Minimal infrastructure drift over time. +- **Code Quality**: High test coverage and compliance with type checking. +- **Performance**: Efficient resource utilization and optimal system performance. + +### Performance Metrics + +- **Time-to-Production Reduction**: Decrease in the time required to deploy new applications or features. +- **Operational Cost Savings**: Reduction in operational expenses due to automation and optimization. +- **Compliance Achievement**: Successful audits and certifications obtained in less time with lower overhead. +- **Practitioner Engagement**: Positive feedback from teams regarding the development environment and tools. + +--- + +## Risk Management + +### Potential Risks and Mitigation Strategies + +1. **Complexity of Multi-Cloud Support** + + - **Risk**: Increased complexity may lead to delays and errors. + - **Mitigation**: Adopt a phased approach, starting with a primary cloud provider before adding others. + +2. **Compliance Changes** + + - **Risk**: Evolving regulations may require significant adjustments. + - **Mitigation**: Establish a compliance team to monitor changes and update policies promptly. + +3. **Resource Constraints** + + - **Risk**: Limited personnel or budget could impact progress. + - **Mitigation**: Prioritize critical components and seek additional resources or adjust timelines as necessary. + +4. **Technological Shifts** + + - **Risk**: Emerging technologies may render parts of the platform obsolete. + - **Mitigation**: Maintain flexibility in the architecture and stay informed about industry trends. + +--- + +## Appendices + +### A. Technical Specifications + +- **Programming Language**: Python 3.10+ +- **Infrastructure as Code Tool**: Pulumi +- **Cloud Providers**: Kubernetes on AWS (initial focus), with Azure, GCP, and bare metal Kubernetes in later phases +- **Type Checking**: Pyright +- **Testing Framework**: pytest +- **Dependency Management**: Poetry + +### B. Compliance Requirements + +- **NIST SP 800-53**: Security and privacy controls +- **FISMA Moderate**: Federal Information Security Management Act compliance +- **ISO 27001**: Information security management standards + +### C. Reference Architectures + +- **AWS Well-Architected Framework** +- **Azure Cloud Adoption Framework** +- **GCP Best Practices for Enterprise Organizations** + +### D. Development Tools + +- **Documentation**: Markdown (following the [Documentation Style Guide](./developer_guide/documentation.md)) +- **Linting**: Flake8 +- **Code Formatting**: Black +- **Version Control**: Github +- **Continuous Integration**: GitHub Actions + +--- + +## Related Documentation + +- [Developer Guide](./developer_guide/README.md) +- [Pulumi Python Development Guide](./developer_guide/pulumi-python.md) +- [Core Module Documentation](./modules/core/README.md) +- [Documentation Style Guide](./developer_guide/documentation.md) +- [Contribution Guidelines](./contribution_guidelines.md) diff --git a/docs/developer_guide/contribution_guidelines.md b/docs/developer_guide/contribution_guidelines.md deleted file mode 100644 index a935b0d..0000000 --- a/docs/developer_guide/contribution_guidelines.md +++ /dev/null @@ -1,378 +0,0 @@ -# Contribution Guidelines - -## Introduction - -Welcome to the Konductor contribution guidelines! This document provides detailed instructions for contributing to the Konductor project, whether you're fixing bugs, adding features, improving documentation, or creating new modules. We value all contributions and want to make the process as transparent and straightforward as possible. - -## Table of Contents - -1. [Code of Conduct](#code-of-conduct) -2. [Getting Started](#getting-started) -3. [Development Environment](#development-environment) -4. [Contribution Workflow](#contribution-workflow) -5. [Documentation Guidelines](#documentation-guidelines) -6. [Testing Requirements](#testing-requirements) -7. [Code Style and Standards](#code-style-and-standards) -8. [Pull Request Process](#pull-request-process) -9. [Issue Guidelines](#issue-guidelines) -10. [Community Engagement](#community-engagement) - -## Code of Conduct - -Our project adheres to a Code of Conduct that establishes expected behavior for all contributors and community members. Please read and follow our [Code of Conduct](../CODE_OF_CONDUCT.md). - -## Getting Started - -### Prerequisites - -Before contributing, ensure you have: - -- Python 3.8 or higher -- Poetry for dependency management -- Pulumi CLI -- Git -- A code editor (VS Code recommended) -- AWS CLI (for AWS module development) -- kubectl (for Kubernetes development) - -### Initial Setup - -1. **Fork the Repository** - ```bash - # Fork via GitHub UI, then clone your fork - git clone https://github.com/YOUR_USERNAME/konductor.git - cd konductor - ``` - -2. **Set Up Development Environment** - ```bash - # Install dependencies - poetry install - - # Activate virtual environment - poetry shell - - # Install pre-commit hooks - pre-commit install - ``` - -## Development Environment - -### Required Tools - -- **VS Code Extensions**: - - Pylance for Python language support - - Python extension for debugging - - YAML extension for configuration files - - Docker extension for container management - -### Configuration Files - -1. **Pyright Configuration** (`pyrightconfig.json`): - ```json - { - "include": ["**/*.py"], - "exclude": ["**/__pycache__/**"], - "reportMissingImports": true, - "pythonVersion": "3.8", - "typeCheckingMode": "strict" - } - ``` - -2. **Poetry Configuration** (`pyproject.toml`): - ```toml - [tool.poetry] - name = "konductor" - version = "0.1.0" - description = "Infrastructure as Code platform" - - [tool.poetry.dependencies] - python = "^3.8" - pulumi = "^3.0.0" - ``` - -## Contribution Workflow - -### 1. Create an Issue - -Before starting work: -- Check existing issues and discussions -- Create a new issue using the appropriate template: - - [Bug Report Template](../contribution_templates/issue_template.md) - - [Feature Request Template](../contribution_templates/feature_request_template.md) - -### 2. Branch Creation - -```bash -#Create a feature branch -git checkout -b feature/issue-number-brief-description - -# For bug fixes -git checkout -b fix/issue-number-brief-description -``` - - -### 3. Development Process - -1. **Write Code** - - Follow [Python Development Standards](../reference/PULUMI_PYTHON.md) - - Use type hints and TypedDict (see [TypedDict Guide](../reference/TypedDict.md)) - - Add tests for new functionality - -2. **Local Testing** - ```bash - # Run type checking - poetry run pyright - - # Run tests - poetry run pytest - - # Run linting - poetry run black . - poetry run isort . - poetry run flake8 . - ``` - -3. **Commit Changes** - ```bash - # Stage changes - git add . - - # Commit with conventional commit message - git commit -m "type(scope): description" - ``` - -### 4. Documentation Updates - -All contributions must include appropriate documentation updates: - -1. **Code Documentation** - - Docstrings for all public functions/classes - - Type annotations - - Inline comments for complex logic - -2. **User Documentation** - - Update relevant user guides - - Add examples if applicable - - Update FAQs if needed - -3. **Developer Documentation** - - Update technical documentation - - Add architecture diagrams if needed - - Update module documentation - -## Documentation Guidelines - -### File Organization - -Follow the documentation structure: - -```bash -docs/ -├── user_guide/ # End-user documentation -├── developer_guide/ # Developer documentation -├── modules/ # Module-specific guides -├── reference/ # Technical references -└── contribution_templates/ # Contribution templates -``` - -### Documentation Standards - -1. **Markdown Formatting** - - Use ATX-style headers (`#` for headers) - - Include table of contents for long documents - - Use code blocks with language identifiers - - Include alt text for images - -2. **Content Guidelines** - - Write in clear, concise language - - Include examples and use cases - - Link to related documentation - - Keep technical accuracy - -3. **Accessibility** - - Use proper heading hierarchy - - Provide alt text for images - - Ensure sufficient color contrast - - Use descriptive link text - -### Example Documentation - -```python -from typing import Dict, Optional - -def update_resource_tags( - resource_id: str, - tags: Dict[str, str], - region: Optional[str] = None -) -> Dict[str, str]: - """Update tags for an AWS resource. - - Args: - resource_id: The ID of the resource to update - tags: Dictionary of tags to apply - region: Optional AWS region (defaults to current) - - Returns: - Dictionary of applied tags - - Raises: - ResourceNotFoundError: If resource doesn't exist - InvalidTagError: If tags are invalid - - Example: - >>> tags = update_resource_tags("vpc-123", {"Environment": "prod"}) - >>> assert tags["Environment"] == "prod" - """ -``` - - -## Testing Requirements - -### Required Tests - -1. **Unit Tests** - - Test individual components - - Use pytest fixtures - - Mock external dependencies - -2. **Integration Tests** - - Test module interactions - - Verify resource creation - - Test configuration handling - -3. **Type Checking** - - Use strict type checking - - Verify all type annotations - -### Example Test Structure - -```python -import pytest -from pulumi import automation as auto - -def test_vpc_creation(): - """Test VPC creation with default configuration.""" - stack = auto.create_stack(...) - - # Deploy resources - result = stack.up() - - # Verify outputs - assert "vpc_id" in result.outputs - assert result.outputs["vpc_id"].value != "" -``` - - -## Code Style and Standards - -### Python Standards - -1. **Type Safety** - - Use type hints for all functions - - Implement TypedDict for configurations - - Enable strict type checking - -2. **Code Organization** - - Follow single responsibility principle - - Use meaningful names - - Keep functions focused and small - -3. **Error Handling** - - Use custom exceptions - - Provide meaningful error messages - - Handle edge cases - -### Example Code Style - -```python -from typing import TypedDict, List - -class NetworkConfig(TypedDict): - vpc_cidr: str - subnet_cidrs: List[str] - -class NetworkManager: - """Manages AWS networking resources.""" - def __init__(self, config: NetworkConfig): - self.config = config - self.validate_config() - - def validate_config(self) -> None: - """Validate network configuration.""" - if not self.is_valid_cidr(self.config["vpc_cidr"]): - raise ValueError(f"Invalid VPC CIDR: {self.config['vpc_cidr']}") -``` - - -## Pull Request Process - -### 1. Prepare Your PR - -- Update your branch with main -- Ensure all tests pass -- Update documentation -- Add changelog entry - -### 2. Submit PR - -Use the [Pull Request Template](../contribution_templates/pull_request_template.md): - -- Link related issues -- Describe changes -- List testing performed -- Note documentation updates - -### 3. Review Process - -- Address reviewer feedback -- Keep PR focused and small -- Maintain clear communication - -### 4. Merge Requirements - -- Passing CI/CD checks -- Approved reviews -- Updated documentation -- Changelog entry - -## Issue Guidelines - -### Creating Issues - -Use appropriate templates: -- [Bug Report Template](../contribution_templates/issue_template.md) -- [Feature Request Template](../contribution_templates/feature_request_template.md) - -### Issue Labels - -- `bug`: Bug reports -- `enhancement`: Feature requests -- `documentation`: Documentation updates -- `good first issue`: Beginner-friendly -- `help wanted`: Community input needed - -## Community Engagement - -### Communication Channels - -- GitHub Issues and Discussions -- Discord Community - -### Getting Help - -1. Check documentation -2. Search existing issues -3. Ask in Discord -4. Create a new issue - -## Conclusion - -Thank you for contributing to Konductor! Your efforts help make the project better for everyone. Remember to: - -- Follow the guidelines -- Write clear documentation -- Test thoroughly -- Engage with the community - -For updates and new features, watch our [GitHub repository](https://github.com/containercraft/konductor). diff --git a/docs/devcontainer.md b/docs/developer_guide/devcontainer.md similarity index 100% rename from docs/devcontainer.md rename to docs/developer_guide/devcontainer.md diff --git a/docs/developer_guide/documentation.md b/docs/developer_guide/documentation.md new file mode 100644 index 0000000..d0f3a74 --- /dev/null +++ b/docs/developer_guide/documentation.md @@ -0,0 +1,434 @@ +# Konductor Documentation Style Guide + +## Introduction + +This style guide establishes standards for creating and maintaining documentation within the Konductor project. It ensures consistency, clarity, and accessibility across all documentation while aligning with the project's technical standards outlined in `PULUMI_PYTHON.md`. + +This guide is intended for all contributors, including developers, documentation writers, and content engineers. By following these guidelines, we ensure that our documentation is user-friendly, maintainable, and helpful to both new and experienced users of the Konductor platform. + +## Table of Contents + +1. [General Principles](#general-principles) +2. [Document Structure](#document-structure) +3. [Writing Style](#writing-style) +4. [Formatting Standards](#formatting-standards) +5. [Code Examples](#code-examples) +6. [Code Commenting and Docstrings](#code-commenting-and-docstrings) +7. [Links and References](#links-and-references) +8. [Images and Diagrams](#images-and-diagrams) +9. [Accessibility Guidelines](#accessibility-guidelines) +10. [Version Control](#version-control) +11. [File Organization](#file-organization) +12. [Documentation Workflow and Maintenance](#documentation-workflow-and-maintenance) +13. [Conclusion](#conclusion) +14. [Related Documents](#related-documents) + +## General Principles + +### Clarity + +- **Know Your Audience**: Write for your audience's knowledge level, which may range from junior developers to experienced platform engineers. +- **Define Technical Terms**: Introduce and define technical terms on their first use. +- **Consistent Terminology**: Use consistent terminology throughout the documentation. +- **Avoid Unnecessary Jargon**: Use simple language whenever possible and avoid unnecessary jargon. + +### Completeness + +- **Comprehensive Coverage**: Include all necessary information to understand the topic fully. +- **Contextual Information**: Provide context for technical concepts to help users understand why and how something works. +- **Link to Related Content**: Reference related documentation, tutorials, or external resources. +- **Troubleshooting Guidance**: Include troubleshooting tips and common issues where applicable. + +### Maintainability + +- **Modular Content**: Keep content modular to facilitate updates and reuse. +- **Use Relative Links**: Use relative links for internal references to maintain link validity across branches and forks. +- **DRY Principle**: Follow the Don't Repeat Yourself principle to avoid duplication. +- **Regular Updates**: Review and update documentation regularly to keep it current. + +### Accessibility + +- **Inclusive Language**: Use inclusive language and avoid bias. +- **Alternative Text**: Provide alternative text for images and diagrams. +- **Clear Structure**: Organize content logically with appropriate headings. +- **Usability**: Ensure that documentation is easy to navigate and understand. + +## Document Structure + +### Required Sections + +1. **Title**: Clear, descriptive title using H1 (`#`). +2. **Introduction**: Brief overview of the document's purpose and scope. +3. **Table of Contents**: For documents longer than three sections. +4. **Prerequisites** (if applicable): Required knowledge, software, or setup before proceeding. +5. **Main Content**: Organized into logical sections with clear headings. +6. **Conclusion** (if applicable): Summary of key points or next steps. +7. **Related Resources**: Links to related documentation, tutorials, or external resources. + +### Header Hierarchy + +Use the following hierarchy for headings: + +```markdown +# Document Title (H1) +## Section Title (H2) +### Sub-section Title (H3) +#### Minor Sub-section Title (H4) +##### Sub-sub-section Title (H5) +``` + +### Metadata Block (Optional) + +Include a metadata block at the beginning of the document if additional context is needed: + +```yaml +--- +title: Document Title +description: Brief description of the document +authors: [Author Name] +date: YYYY-MM-DD +version: 0.0.1 +--- +``` + +## Writing Style + +### Voice and Tone + +- **Active Voice**: Use active voice to make sentences clear and direct. +- **Conciseness**: Be direct and concise; avoid unnecessary words. +- **Professional and Friendly Tone**: Maintain a professional tone while being approachable. +- **Present Tense**: Write in the present tense when possible. + +### Language + +- **Consistency**: Use consistent terminology and style throughout the documentation. +- **Imperative Mood**: Use the imperative mood for instructions (e.g., "Install the package by running..."). +- **Avoid Ambiguity**: Be specific to avoid confusion. + +### Paragraphs + +- **Focused Paragraphs**: Keep paragraphs focused on a single idea or topic. +- **Short Paragraphs**: Use short paragraphs (3-5 sentences) to enhance readability. +- **Transitions**: Include transition sentences or phrases to connect ideas between paragraphs and sections. + +### Lists + +- **Bullet Points**: Use bullet points for unordered lists or items that do not require a specific order. +- **Numbered Lists**: Use numbered lists for sequential steps or when order is important. +- **Parallel Structure**: Ensure that list items have a parallel grammatical structure. + +## Formatting Standards + +### Text Formatting + +- **Bold**: Use bold text for emphasis, UI elements (e.g., button names), and important terms. +- *Italic*: Use italics when introducing new terms or for light emphasis. +- `Code`: Use inline code formatting for code snippets, file names, commands, and configuration keys. +- ***Bold Italic***: Avoid using bold italic text unless necessary. + +### Headings + +- **Capitalization**: Use sentence case for headings (capitalize only the first word and proper nouns). +- **Avoid Punctuation**: Do not include periods at the end of headings. + +### Code Blocks + +- **Syntax Highlighting**: Use triple backticks with the language identifier for syntax highlighting. +- **Description**: Include a brief description or comment explaining the purpose of the code. +- **Line Numbers**: For longer code snippets, consider adding line numbers for reference. +- **Avoid Horizontal Scrolling**: Ensure code fits within the page width; break long lines if necessary. + +Example: + +```python +# Example code block: Function to calculate factorial +def factorial(n: int) -> int: + """Calculate the factorial of a non-negative integer n.""" + if n == 0: + return 1 + else: + return n * factorial(n - 1) +``` + +### Tables + +- **Usage**: Use tables for presenting structured data that is best displayed in rows and columns. +- **Header Row**: Include a header row to label columns. +- **Alignment**: Align columns consistently; left-align text for readability. + +Example: + +| Configuration Option | Type | Description | +|----------------------|--------|--------------------------------------| +| `enabled` | bool | Whether the feature is enabled | +| `timeout` | int | Timeout duration in seconds | +| `retry_attempts` | int | Number of retry attempts on failure | + +## Code Examples + +### General Guidelines + +- **Relevance**: Ensure examples are relevant to the context and demonstrate best practices. +- **Simplicity**: Keep examples simple and focused on the key concept. +- **Clarity**: Include comments explaining key concepts or steps. +- **Consistency**: Use consistent code formatting and style, following the project's coding standards. + +### Example Structure + +- **Imports**: Include necessary imports for the code to be self-contained. +- **Docstrings and Comments**: Use docstrings and inline comments to explain the code. +- **Meaningful Names**: Use descriptive variable and function names. + +Example: + +```python +from typing import TypedDict, Optional + +class ConfigExample(TypedDict): + """Example configuration structure. + + Attributes: + name: The name of the resource. + enabled: Indicates if the resource is enabled. + """ + name: str + enabled: bool + +def example_function(config: ConfigExample) -> None: + """Process the configuration and print the resource status. + + Args: + config: A configuration dictionary containing 'name' and 'enabled'. + """ + if config.get("enabled", True): + print(f"Resource '{config['name']}' is enabled.") + else: + print(f"Resource '{config['name']}' is disabled.") +``` + +## Code Commenting and Docstrings + +### Docstring Standards + +- **Purpose**: Every module, class, and function/method should have a docstring that describes its purpose and usage. +- **Style**: Use triple double-quotes (`"""`) for docstrings. +- **Content**: + - **Modules**: Describe the module's purpose and main functionalities. + - **Classes**: Provide an overview of the class, its responsibilities, and any important details. + - **Functions/Methods**: Explain what the function does, its parameters, return values, exceptions raised, and any side effects. + +### Docstring Format + +- **Summary Line**: Begin with a concise summary of the object. +- **Blank Line**: Add a blank line after the summary. +- **Details**: Provide more detailed information if necessary. +- **Args**: List and describe each parameter. +- **Returns**: Describe the return value. +- **Raises**: List any exceptions that the function may raise. + +Example: + +```python +def connect_to_database(host: str, port: int) -> DatabaseConnection: + """Establish a connection to the database. + + Args: + host: The hostname of the database server. + port: The port number on which the database is listening. + + Returns: + A DatabaseConnection object representing the connection. + + Raises: + ConnectionError: If the connection cannot be established. + """ + # Function implementation +``` + +### Inline Comments + +- **Usage**: Use inline comments sparingly to explain complex logic or important details not immediately apparent from the code. +- **Placement**: Place the comment above the code line or at the end of the line if brief. +- **Clarity**: Ensure comments are clear and add value beyond what the code expresses. + +Example: + +```python +# Calculate the Fibonacci sequence up to n terms +def fibonacci(n: int) -> List[int]: + sequence = [0, 1] + for _ in range(2, n): + # Append the sum of the last two numbers + sequence.append(sequence[-1] + sequence[-2]) + return sequence +``` + +### Code Formatting + +- **Consistency**: Follow the project's coding standards as outlined in `PULUMI_PYTHON.md`. +- **PEP 8 Compliance**: Adhere to the PEP 8 style guide for Python code. +- **Type Annotations**: Use type hints consistently. +- **Linting**: Ensure code passes linting checks (e.g., using `flake8` or `pylint`). + +## Links and References + +### Internal Links + +- **Relative Paths**: Use relative paths for internal links to ensure they work across branches and forks. +- **Section Links**: Link to specific sections using the heading text converted to lowercase and spaces replaced with hyphens (e.g., `#section-title`). +- **Validity**: Regularly check links to ensure they are valid and not broken. + +Example: + +- `[Introduction](#introduction)` +- `[PULUMI_PYTHON.md](../reference/PULUMI_PYTHON.md)` + +### External Links + +- **Descriptive Text**: Use link text that describes the content of the linked page. +- **Target Stability**: Prefer linking to stable URLs that are unlikely to change. +- **Notes on External Dependencies**: If the link is to an external dependency or resource, consider including the version or date accessed. + +Example: + +- `[Pulumi Documentation](https://www.pulumi.com/docs/)` + +### Cross-References + +- **Consistent Terminology**: Use the same terms when referring to concepts or components. +- **Glossary Terms**: Link to definitions of terms in a glossary if available. +- **Related Documentation**: Reference related guides or tutorials to provide additional context. + +## Images and Diagrams + +### Requirements + +- **Format**: Use SVG format for diagrams when possible to ensure scalability. +- **Alternative Text**: Provide alt text that describes the image for accessibility. +- **Resolution**: Ensure images are high-resolution but optimized for web to reduce file size. +- **File Naming**: Use descriptive file names in lowercase with hyphens. + +### Including Images + +- **Placement**: Place images in an `images` or `assets` subdirectory relevant to the document. +- **Markdown Syntax**: Use the following syntax to include images: + + ```markdown + ![Alt text describing the image](./images/diagram.svg) + ``` + +### Captions + +- **Descriptive Captions**: Include a caption below the image if additional explanation is needed. +- **Figure Numbers**: Number figures sequentially if the document contains multiple images. + +Example: + +```markdown +![Deployment Architecture Diagram](./images/deployment-architecture.svg) + +*Figure 1: High-level deployment architecture of the Konductor platform.* +``` + +## Accessibility Guidelines + +### Text Content + +- **Color Contrast**: Ensure sufficient color contrast between text and background. +- **Avoid Color Reliance**: Do not rely solely on color to convey information. +- **Semantic Markup**: Use appropriate heading levels and list structures. + +### Navigation + +- **Logical Structure**: Organize content with a clear and logical structure. +- **Descriptive Links**: Use descriptive link text that makes sense out of context. +- **Keyboard Accessibility**: Ensure interactive elements can be accessed via keyboard navigation. + +### Media + +- **Alt Text**: Provide alternative text for images that conveys the same information. +- **Transcripts and Captions**: Provide transcripts for audio content and captions for videos. +- **Accessible Tables**: Use table headers and scope attributes to make tables accessible. + +## Version Control + +### Commit Messages + +- **Descriptive Messages**: Write clear, descriptive commit messages that explain the changes made. +- **Issue References**: Reference related issue numbers in the commit message. +- **Conventional Commits**: Follow the [Conventional Commits](https://www.conventionalcommits.org/) specification. + +Example: + +``` +docs(developer_guide): add code commenting standards to documentation style guide + +* Include section on code commenting and docstrings +* Update examples with docstrings +* Align with PULUMI_PYTHON.md coding standards + +Fixes: #456 +``` + +### Branching + +- **Feature Branches**: Create feature branches for substantial changes or additions. +- **Naming Conventions**: Use prefixes like `docs/` for documentation branches (e.g., `docs/update-style-guide`). +- **Pull Requests**: Submit pull requests for review before merging changes into the main branch. +- **Synchronization**: Keep documentation in sync with code changes to maintain consistency. + +## File Organization + +### Naming Conventions + +- **Lowercase with Hyphens**: Use lowercase letters with hyphens to separate words (e.g., `getting-started.md`). +- **Descriptive Names**: Choose descriptive but concise file names that reflect the content. +- **File Extensions**: Use `.md` for Markdown files. + +### Directory Structure + +- **Hierarchy**: Follow the established directory hierarchy of the project. +- **Grouping**: Group related documents in subdirectories (e.g., `developer_guide`, `user_guide`). +- **README Files**: Include `README.md` files in directories to provide an overview and navigation. +- **Clean Organization**: Avoid clutter by removing outdated or unnecessary files. + +### File Templates + +- **Standard Templates**: Use standard templates for common document types (e.g., guides, tutorials). +- **Consistency**: Maintain consistent structure and formatting across documents. +- **Updating Templates**: Update templates as needed to reflect changes in standards or practices. + +## Documentation Workflow and Maintenance + +### Review Process + +- **Peer Review**: Have documentation reviewed by at least one other team member. +- **Feedback Incorporation**: Address feedback and make necessary revisions. +- **Approval**: Obtain necessary approvals before merging changes. + +### Updating Documentation + +- **Version Updates**: Update documentation when code changes affect existing content. +- **Deprecation Notices**: Clearly indicate when features are deprecated and provide alternatives. +- **Changelog**: Maintain a changelog of significant documentation updates. + +### Automation Tools + +- **Continuous Integration**: Use CI tools to check for broken links, spelling errors, and formatting issues. +- **Linting**: Implement Markdown linters to enforce style guidelines. +- **Documentation Generation**: Utilize tools to generate API documentation from code comments where applicable. + +## Conclusion + +By adhering to this documentation style guide, we ensure that the Konductor project's documentation is consistent, clear, and accessible. This facilitates better understanding among users and contributors, aids in onboarding new team members, and enhances the overall quality and maintainability of the project. + +For questions or suggestions regarding this style guide, please reach out to the documentation team or submit an issue on the project's repository. + +## Related Documents + +- [Developer Guide](./README.md) +- [Pulumi Python Development Standards (`PULUMI_PYTHON.md`)](../reference/PULUMI_PYTHON.md) +- [TypedDict Reference Guide](../reference/TypedDict.md) +- [Contribution Guidelines](../contribution_guidelines.md) +- [Style Guide](../reference/style_guide.md) diff --git a/docs/developer_guide/konductor_developer_guide.md b/docs/developer_guide/konductor_developer_guide.md deleted file mode 100644 index 4b90366..0000000 --- a/docs/developer_guide/konductor_developer_guide.md +++ /dev/null @@ -1,738 +0,0 @@ -# Konductor Developer Guide - -## Introduction - -Welcome to the comprehensive developer guide for Konductor! This guide provides detailed information for developers who want to contribute to or extend the Konductor Infrastructure as Code (IaC) platform. Whether you're fixing bugs, adding features, or creating new modules, you'll find everything you need to understand our development practices and standards. - -## Table of Contents - -1. [Development Environment](#development-environment) - - [Prerequisites](#prerequisites) - - [Environment Setup](#environment-setup) - - [Development Tools](#development-tools) - -2. [Core Technologies](#core-technologies) - - [Pulumi Overview](#pulumi-overview) - - [Poetry for Dependency Management](#poetry-for-dependency-management) - - [Type Safety with TypedDict](#type-safety-with-typeddict) - - [Static Type Checking](#static-type-checking) - -3. [Project Structure](#project-structure) - - [Directory Layout](#directory-layout) - - [Module Organization](#module-organization) - - [Configuration Management](#configuration-management) - -4. [Development Standards](#development-standards) - - [Code Style Guidelines](#code-style-guidelines) - - [Type Annotations](#type-annotations) - - [Documentation Requirements](#documentation-requirements) - - [Testing Requirements](#testing-requirements) - -5. [Module Development](#module-development) - - [Module Architecture](#module-architecture) - - [Creating New Modules](#creating-new-modules) - - [Module Testing](#module-testing) - - [Module Documentation](#module-documentation) - -6. [Testing and Validation](#testing-and-validation) - - [Unit Testing](#unit-testing) - - [Integration Testing](#integration-testing) - - [Type Checking](#type-checking) - - [Continuous Integration](#continuous-integration) - -7. [Documentation Guidelines](#documentation-guidelines) - - [Code Documentation](#code-documentation) - - [Module Documentation](#module-documentation-1) - - [API Documentation](#api-documentation) - - [Example Documentation](#example-documentation) - -8. [Best Practices](#best-practices) - - [Code Organization](#code-organization) - - [Error Handling](#error-handling) - - [Configuration Management](#configuration-management-1) - - [Resource Management](#resource-management) - -9. [Troubleshooting and Support](#troubleshooting-and-support) - - [Common Issues](#common-issues) - - [Getting Help](#getting-help) - - [Community Resources](#community-resources) - -## Development Environment - -### Prerequisites - -Before starting development, ensure you have: - -- Python 3.8 or higher -- Poetry for dependency management -- Pulumi CLI -- Git -- A code editor (VS Code recommended) -- AWS CLI (for AWS module development) -- kubectl (for Kubernetes development) - -### Environment Setup - -1. **Clone the Repository**: - ```bash - git clone https://github.com/containercraft/konductor.git - cd konductor - ``` - -2. **Install Dependencies**: - ```bash - poetry install - ``` - -3. **Activate Virtual Environment**: - ```bash - poetry shell - ``` - -4. **Configure Development Tools**: - ```bash - # Configure Pulumi - pulumi login - - # Set up pre-commit hooks - pre-commit install - ``` - -### Development Tools - -- **VS Code Extensions**: - - Pylance for Python language support - - Python extension for debugging - - YAML extension for configuration files - - Docker extension for container management - -- **Configuration Files**: - ```json:pyrightconfig.json - { - "include": ["**/*.py"], - "exclude": ["**/__pycache__/**"], - "reportMissingImports": true, - "pythonVersion": "3.8", - "typeCheckingMode": "strict" - } - ``` - -## Core Technologies - -### Pulumi Overview - -Pulumi is our primary IaC framework, chosen for its: -- Native Python support -- Strong type system integration -- Multi-cloud capabilities -- State management features - -### Poetry for Dependency Management - -We use Poetry to: -- Manage project dependencies -- Create reproducible builds -- Handle virtual environments -- Package distribution - -Example `pyproject.toml`: -```toml -[tool.poetry] -name = "konductor" -version = "0.1.0" -description = "Infrastructure as Code platform" - -[tool.poetry.dependencies] -python = "^3.8" -pulumi = "^3.0.0" -pulumi-aws = "^5.0.0" -``` - -### Type Safety with TypedDict - -TypedDict is central to our configuration management: - -```python -from typing import TypedDict, List - -class ContainerPort(TypedDict): - containerPort: int - protocol: str - -class Container(TypedDict): - name: str - image: str - ports: List[ContainerPort] -``` - -### Static Type Checking - -We enforce strict type checking using Pyright: - -```bash -# Run type checking -poetry run pyright - -# Configure VS Code for real-time type checking -{ - "python.analysis.typeCheckingMode": "strict" -} -``` - -## Project Structure - -### Directory Layout - -``` -konductor/ -├── __main__.py -├── core/ -│ ├── __init__.py -│ ├── config.py -│ └── utils.py -└── modules/ -│ ├── aws/ -│ └── kubernetes/ -├── docs/ -└── tests/ -``` - -### Module Organization - -Each module follows a standard structure: - -``` -modules// -├── __init__.py -├── types.py -├── deploy.py -├── config.py -└── README.md -``` - -### Configuration Management - -Configuration hierarchy: -1. Default module configurations -2. User-provided configurations -3. Environment-specific overrides - -## Development Standards - -### Code Style Guidelines - -We follow PEP 8 with additional requirements: - -```python -# Good -def create_deployment( - name: str, - replicas: int, - container: Container -) -> Deployment: - """Create a Kubernetes deployment. - - Args: - name: The deployment name - replicas: Number of replicas - container: Container configuration - - Returns: - The created deployment - """ - return Deployment(...) - -# Bad - Missing type hints and docstring -def create_deployment(name, replicas, container): - return Deployment(...) -``` - -### Type Annotations - -All code must use type hints: - -```python -from typing import Dict, List, Optional - -def get_resource_tags( - environment: str, - additional_tags: Optional[Dict[str, str]] = None -) -> Dict[str, str]: - tags = {"Environment": environment} - if additional_tags: - tags.update(additional_tags) - return tags -``` - -### Documentation Requirements - -Required documentation elements: -- Module overview -- Function/class docstrings -- Type annotations -- Usage examples -- Configuration options - -### Testing Requirements - -All code must include: -- Unit tests -- Integration tests (where applicable) -- Type checking validation -- Documentation tests - -## Module Development - -### Module Architecture - -Modules should follow the Single Responsibility Principle: - -```python -# Good - Single responsibility -class AwsNetworking: - """Manages AWS networking resources.""" - - def create_vpc(self) -> pulumi.Output[str]: - """Create a VPC.""" - pass - - def create_subnet(self) -> pulumi.Output[str]: - """Create a subnet.""" - pass - -# Bad - Mixed responsibilities -class AwsResources: - """Manages various AWS resources.""" - - def create_vpc(self) -> str: - pass - - def create_database(self) -> str: - pass -``` - -### Creating New Modules - -Follow these steps to create a new module: - -1. Create module directory structure -2. Define TypedDict configurations -3. Implement core functionality -4. Add documentation -5. Write tests -6. Submit for review - -### Module Testing - -Example test structure: - -```python -import pytest -from pulumi import automation as auto - -def test_vpc_creation(): - """Test VPC creation with default configuration.""" - stack = auto.create_stack(...) - - # Deploy resources - result = stack.up() - - # Verify outputs - assert "vpc_id" in result.outputs - assert result.outputs["vpc_id"].value != "" -``` - -### Module Documentation - -Required module documentation: - -1. Overview and purpose -2. Installation instructions -3. Configuration options -4. Usage examples -5. API reference -6. Troubleshooting guide - -## Testing and Validation - -### Unit Testing - -We use pytest for unit testing: - -```python -import pytest -from konductor.core.utils import merge_configurations - -def test_merge_configurations(): - """Test configuration merging logic.""" - base = {"name": "test", "replicas": 1} - override = {"replicas": 2} - - result = merge_configurations(base, override) - - assert result["name"] == "test" - assert result["replicas"] == 2 -``` - -### Integration Testing - -Integration tests verify module interactions: - -```python -from pulumi import automation as auto -from typing import Generator -import pytest - -@pytest.fixture -def pulumi_stack() -> Generator[auto.Stack, None, None]: - """Create a test stack for integration testing.""" - stack = auto.create_stack( - stack_name="integration-test", - project_name="konductor-test" - ) - yield stack - stack.destroy() - stack.workspace.remove_stack("integration-test") - -def test_aws_module_integration(pulumi_stack: auto.Stack): - """Test AWS module integration with core components.""" - # Deploy test infrastructure - result = pulumi_stack.up() - - # Verify resource creation - assert "vpc_id" in result.outputs - assert "subnet_ids" in result.outputs -``` - -### Type Checking - -Run type checking as part of validation: - -```bash -# Run type checking with detailed output -poetry run pyright --verbose - -# Check specific module -poetry run pyright pulumi/modules/aws -``` - -### Continuous Integration - -Our CI pipeline includes: - -1. **Code Quality Checks**: - ```yaml - steps: - - name: Code Quality - run: | - poetry run black --check . - poetry run isort --check-only . - poetry run flake8 . - ``` - -2. **Type Checking**: - ```yaml - steps: - - name: Type Check - run: poetry run pyright - ``` - -3. **Tests**: - ```yaml - steps: - - name: Run Tests - run: poetry run pytest --cov - ``` - -## Documentation Guidelines - -### Code Documentation - -Follow these docstring conventions: - -```python -from typing import Dict, Optional - -def update_resource_tags( - resource_id: str, - tags: Dict[str, str], - region: Optional[str] = None -) -> Dict[str, str]: - """Update tags for an AWS resource. - - Args: - resource_id: The ID of the resource to update - tags: Dictionary of tags to apply - region: Optional AWS region (defaults to current) - - Returns: - Dictionary of applied tags - - Raises: - ResourceNotFoundError: If resource doesn't exist - InvalidTagError: If tags are invalid - - Example: - >>> tags = update_resource_tags("vpc-123", {"Environment": "prod"}) - >>> assert tags["Environment"] == "prod" - """ - # Implementation -``` - -### Module Documentation - -Each module must include: - -1. **README.md**: - ```markdown - # AWS Module - - ## Overview - Manages AWS infrastructure resources. - - ## Installation - ```bash - poetry add konductor-aws - ``` - - ## Usage - ```python - from konductor.modules.aws import AwsNetworking - - networking = AwsNetworking(...) - vpc = networking.create_vpc(...) - ``` - - ## Configuration - | Parameter | Type | Description | Default | - |-----------|------|-------------|---------| - | region | str | AWS region | us-west-2 | - ``` - -2. **API Documentation**: - ```python - class AwsNetworking: - """AWS networking resource management. - - Provides functionality for creating and managing AWS networking - resources including VPCs, subnets, and security groups. - - Attributes: - region: The AWS region for resource creation - tags: Default tags for all resources - """ - ``` - -### Example Documentation - -Provide clear, runnable examples: - -```python -from konductor.modules.aws import AwsNetworking -from konductor.core.config import NetworkConfig - -# Configuration -network_config = NetworkConfig( - vpc_cidr="10.0.0.0/16", - subnet_cidrs=["10.0.1.0/24", "10.0.2.0/24"], - availability_zones=["us-west-2a", "us-west-2b"] -) - -# Create networking resources -networking = AwsNetworking(network_config) -vpc = networking.create_vpc() -subnets = networking.create_subnets() -``` - -## Best Practices - -### Code Organization - -1. **Separation of Concerns**: - ```python - # Good - Separate configuration and implementation - class NetworkConfig(TypedDict): - vpc_cidr: str - subnet_cidrs: List[str] - - class NetworkManager: - def __init__(self, config: NetworkConfig): - self.config = config - - # Bad - Mixed configuration and implementation - class Network: - def __init__(self, vpc_cidr: str, subnet_cidrs: List[str]): - self.create_vpc(vpc_cidr) - self.create_subnets(subnet_cidrs) - ``` - -2. **Resource Organization**: - ```python - # Good - Logical resource grouping - class AwsNetworking: - def __init__(self, config: NetworkConfig): - self.vpc = self._create_vpc() - self.subnets = self._create_subnets() - self.security_groups = self._create_security_groups() - - # Bad - No clear organization - class AwsResources: - def create_stuff(self): - self.vpc = create_vpc() - self.database = create_database() - self.subnets = create_subnets() - ``` - -### Error Handling - -1. **Custom Exceptions**: - ```python - class ResourceError(Exception): - """Base exception for resource operations.""" - pass - - class ResourceNotFoundError(ResourceError): - """Raised when a resource cannot be found.""" - pass - - def get_resource(resource_id: str) -> Resource: - try: - return fetch_resource(resource_id) - except ApiError as e: - raise ResourceNotFoundError(f"Resource {resource_id} not found") from e - ``` - -2. **Graceful Degradation**: - ```python - from typing import Optional - - def get_resource_tags( - resource_id: str, - default: Optional[Dict[str, str]] = None - ) -> Dict[str, str]: - """Get resource tags with fallback to defaults.""" - try: - return fetch_resource_tags(resource_id) - except ResourceError: - return default or {} - ``` - -### Configuration Management - -1. **Configuration Validation**: - ```python - from pydantic import BaseModel, Field - - class VpcConfig(BaseModel): - cidr_block: str = Field(..., regex=r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/\d{1,2}$") - enable_dns: bool = True - tags: Dict[str, str] = Field(default_factory=dict) - ``` - -2. **Environment-Specific Configuration**: - ```python - def load_config(environment: str) -> Dict[str, Any]: - """Load environment-specific configuration.""" - base_config = load_yaml("config/base.yaml") - env_config = load_yaml(f"config/{environment}.yaml") - return deep_merge(base_config, env_config) - ``` - -### Resource Management - -1. **Resource Cleanup**: - ```python - class ResourceManager: - def __init__(self): - self.resources: List[Resource] = [] - - def add_resource(self, resource: Resource): - self.resources.append(resource) - - def cleanup(self): - """Clean up all managed resources.""" - for resource in reversed(self.resources): - try: - resource.destroy() - except ResourceError: - logger.exception(f"Failed to cleanup {resource.id}") - ``` - -2. **Resource Dependencies**: - ```python - class NetworkStack: - def __init__(self): - self.vpc = self._create_vpc() - # Subnets depend on VPC - self.subnets = self._create_subnets(self.vpc.id) - # Route tables depend on VPC and subnets - self.route_tables = self._create_route_tables( - self.vpc.id, - self.subnets - ) - ``` - -## Troubleshooting and Support - -### Common Issues - -1. **Type Checking Errors**: - ```python - # Error: Parameter "config" missing required TypedDict key "region" - def deploy(config: AwsConfig): # Missing region in config - pass - - # Fix: Provide all required keys - config: AwsConfig = { - "region": "us-west-2", - "tags": {"Environment": "prod"} - } - ``` - -2. **Resource Creation Failures**: - ```python - try: - vpc = create_vpc(config) - except pulumi.ResourceError as e: - # Check for common issues - if "InvalidCidrBlock" in str(e): - logger.error("Invalid CIDR block specified") - elif "QuotaExceeded" in str(e): - logger.error("AWS account quota exceeded") - raise - ``` - -### Getting Help - -1. **Community Support**: - - GitHub Issues: [Konductor Issues](https://github.com/containercraft/konductor/issues) - - Discord Community: [Join Discord](https://discord.gg/Jb5jgDCksX) - - Stack Overflow: Tag `konductor` - -2. **Documentation Resources**: - - [User Guide](../user_guide/README.md) - - [API Reference](../reference/README.md) - - [Module Documentation](../modules/README.md) - -### Community Resources - -1. **Contributing**: - - [Contribution Guidelines](../contribution_guidelines.md) - - [Code of Conduct](../CODE_OF_CONDUCT.md) - - [Development Setup](../getting_started.md) - -2. **Learning Resources**: - - [Tutorial Series](../tutorials/README.md) - - [Example Projects](../examples/README.md) - - [Best Practices Guide](../reference/best_practices.md) - -## Conclusion - -This developer guide provides a comprehensive overview of developing with Konductor. Remember to: - -- Follow type safety practices -- Write comprehensive tests -- Document your code -- Contribute to the community - -For updates and new features, watch our [GitHub repository](https://github.com/containercraft/konductor) diff --git a/docs/developer_guide/module_refactoring.md b/docs/developer_guide/module_refactoring.md deleted file mode 100644 index 295ed90..0000000 --- a/docs/developer_guide/module_refactoring.md +++ /dev/null @@ -1,229 +0,0 @@ -# Module Refactoring Guide - -## Overview -This document outlines the standardized approach for refactoring Pulumi infrastructure modules to follow Python best practices while maintaining infrastructure-as-code principles. The goal is to create a consistent, maintainable, and testable module structure across the codebase. - -## Core Principles - -### 1. Clear Module Interface -- Each module must expose its public API through `__init__.py` -- Use `__all__` to explicitly declare public interfaces -- Provide type hints for all public interfaces -- Document the module's purpose and usage in the module docstring - -### 2. Separation of Concerns -Modules should separate functionality into distinct files: -- `__init__.py`: Public API and entry points -- `types.py`: Type definitions and configuration classes -- Resource-specific implementation files (e.g., `provider.py`, `resources.py`) -- No direct resource creation in `__init__.py` - -### 3. Object-Oriented Design -- Use classes to encapsulate related functionality -- Prefer composition over inheritance -- Implement clear class responsibilities -- Use properties for computed or lazy-loaded values - -### 4. Type Safety -- Use type hints consistently throughout the codebase -- Leverage Pydantic models for configuration validation -- Define clear interfaces using Protocol classes where appropriate -- Use TypedDict for structured dictionary types - -## Standard Module Structure - -```bash -module// -├── __init__.py -├── types.py -├── provider.py -├── resources.py -├── [component_name].py -└── README.md -``` - -### File Responsibilities - -#### __init__.py -- Expose the public API -- Define the module's entry points -- Import all public interfaces and submodules -- Use `__all__` to manage the public API - -```python -""" -Module docstring describing the purpose and functionality of the module. -""" -from typing import List, Optional, Tuple -from .types import ModuleConfig -from .resources import ResourceManager - -__all__ = [ - "ModuleConfig", - "ResourceManager", - "create_infrastructure" -] - -def create_infrastructure( - config: ModuleConfig, - dependencies: Optional[List[pulumi.Resource]] = None -) -> Tuple[str, pulumi.Resource, Dict[str, Any]]: - """ - Create infrastructure with the given configuration. - """ - pass -``` - -#### types.py - -```python -from typing import TypedDict, Optional -from pydantic import BaseModel - -class ModuleConfig(BaseModel): - """ - Module configuration. - """ - pass -``` - -#### resources.py - -```python -from pulumi import ProviderResource - -class ResourceManager: - """ - Manages the creation and configuration of module resources. - """ - def __init__(self, provider: ProviderResource): - self.provider = provider - - def create_resource(self, config: ModuleConfig) -> pulumi.Resource: - """ - Create a resource with the given configuration. - """ - pass -``` - -## Implementation Requirements - -### 1. Configuration Management -- Use Pydantic models for configuration validation -- Support merging of user configs with defaults -- Validate configuration at initialization time -- Support environment variable overrides where appropriate - -### 2. Resource Management -- Implement idempotent resource creation -- Handle resource dependencies explicitly -- Support resource tagging and metadata -- Implement proper cleanup and error handling - -### 3. Provider Integration -- Abstract provider-specific details -- Support multiple provider configurations -- Handle provider authentication securely -- Support cross-provider dependencies - -### 4. Testing Support -- Design classes for testability -- Support mocking of external dependencies -- Enable unit testing of configuration -- Support integration testing of resource creation - -## Migration Process - -1. **Analysis Phase** - - Review existing module functionality - - Identify public interfaces - - Map resource dependencies - - Document current configuration options - -2. **Refactoring Phase** - - Create new file structure - - Implement type definitions - - Create resource management classes - - Migrate existing functionality - -3. **Testing Phase** - - Write unit tests - - Verify existing functionality - - Test error conditions - - Validate configuration handling - -4. **Documentation Phase** - - Update module documentation - - Add docstrings - - Create usage examples - - Document breaking changes - -## Best Practices - -### Code Organization -- Group related functionality into classes -- Use private methods for implementation details -- Implement clear error handling -- Follow PEP 8 style guidelines - -### Documentation -- Include docstrings for all public interfaces -- Document configuration options -- Provide usage examples -- Document any breaking changes - -### Error Handling -- Use custom exception classes -- Provide meaningful error messages -- Handle resource creation failures -- Implement proper cleanup - -### Testing -- Write unit tests for configuration -- Test resource creation logic -- Implement integration tests -- Test error conditions - -## Breaking Changes -When refactoring modules, maintain backward compatibility: -- Keep existing entry points functional -- Support old configuration formats -- Document migration paths -- Version breaking changes appropriately - -## Example Implementation -See the AWS module implementation as a reference: -- `modules/aws/__init__.py` -- `modules/aws/types.py` -- `modules/aws/resources.py` -- `modules/aws/provider.py` - -## Validation Checklist - -- [ ] Clear public API in `__init__.py` -- [ ] Type definitions in `types.py` -- [ ] Resource management classes implemented -- [ ] Configuration validation -- [ ] Error handling -- [ ] Documentation -- [ ] Unit tests -- [ ] Integration tests -- [ ] Breaking changes documented -- [ ] Migration guide provided - -## Next Steps - -1. Complete AWS module refactoring -2. Review and validate changes -3. Create test suite -4. Document changes -5. Apply pattern to remaining modules -6. Validate full codebase -7. Update global documentation - -## References - -- [Python Package Structure](https://docs.python.org/3/tutorial/modules.html) -- [Type Hints PEP 484](https://www.python.org/dev/peps/pep-0484/) -- [Pulumi Architecture](https://www.pulumi.com/docs/intro/concepts/how-pulumi-works/) -- [Python Style Guide](https://www.python.org/dev/peps/pep-0008/) diff --git a/docs/developer_guide/modules/core/README.md b/docs/developer_guide/modules/core/README.md deleted file mode 100644 index 0f0f0a0..0000000 --- a/docs/developer_guide/modules/core/README.md +++ /dev/null @@ -1,351 +0,0 @@ -# Konductor Core Module Documentation - -## Introduction - -The **Konductor Core Module** serves as the foundational framework for Infrastructure as Code (IaC) using Pulumi and Python. It provides a robust, type-safe, and maintainable architecture that enables teams to work independently while ensuring consistency and compliance across infrastructure deployments. By abstracting common IaC patterns, the core module makes your codebase more DRY (Don't Repeat Yourself), reusable, and dynamic. - -This documentation is intended for a wide audience, including junior, senior, and principal developers, DevOps practitioners, executive leadership, and the open-source community. - -## Table of Contents - -- [Architecture Overview](#architecture-overview) -- [Core Module Components](#core-module-components) - - [Directory Structure](#directory-structure) - - [Design Philosophy](#design-philosophy) -- [Detailed Component Descriptions](#detailed-component-descriptions) - - [`__init__.py`](#__init__py) - - [`config.py`](#configpy) - - [Why Is Configuration Complex?](#why-is-configuration-complex) - - [`deployment.py`](#deploymentpy) - - [`metadata.py`](#metadatapy) - - [`resource_helpers.py`](#resource_helperspy) - - [`types.py`](#typespy) - - [`utils.py`](#utilspy) -- [Kubernetes Module Centralized Version Control](#kubernetes-module-centralized-version-control) -- [Main Entry Point `__main__.py`](#main-entry-point-__main__py) -- [When to Use Core Module vs. Module-Specific Code](#when-to-use-core-module-vs-module-specific-code) -- [Best Practices](#best-practices) -- [Development Guidelines](#development-guidelines) -- [Conclusion](#conclusion) - -## Architecture Overview - -### Directory Structure - -The core module is organized as follows: - -``` -core/ -├── __init__.py # Module initialization and exports -├── config.py # Advanced configuration management system -├── deployment.py # Deployment orchestration and lifecycle management -├── metadata.py # Global metadata and tagging utilities -├── resource_helpers.py # Common resource creation and management functions -├── types.py # Shared type definitions and schemas -└── utils.py # General utility functions -``` - -### Design Philosophy - -The core module implements several key principles: - -1. **Separation of Concerns**: Each component has a single, well-defined responsibility. -2. **Type Safety**: Comprehensive type checking throughout the codebase to prevent runtime errors. -3. **Configuration Isolation**: Modules can define their own configuration schemas independently. -4. **Resource Abstraction**: Centralizes common resource patterns for consistency and reuse. -5. **Compliance Integration**: Built-in support for security and compliance requirements. -6. **Team Independence**: Enables different teams to maintain independent configurations without affecting others. - -## Detailed Component Descriptions - -### `__init__.py` - -This file initializes the core module and exports essential classes and functions for external use. It serves as the entry point for the module, making it easier for other parts of the program to import and utilize core functionalities. - -### `config.py` - -**Purpose**: Manages the complex configuration hierarchy used across the entire IaC codebase. - -**Key Features**: - -- **Layered Configuration**: Merges configurations from global defaults, stack-specific settings, environment variables, and module-specific overrides. -- **Validation**: Ensures that configurations meet predefined schemas and compliance requirements. -- **Flexibility**: Supports multiple deployment environments and scenarios. -- **Team Independence**: Allows each module team to maintain independent configuration specifications. - -#### Why Is Configuration Complex? - -The configuration system is intentionally complex to accommodate: - -- **Multi-Team Development**: Enables teams to define their own configuration schemas without impacting others. -- **Version Control**: Each module maintains its own application versioning, allowing for independent updates. -- **Compliance Requirements**: Centralized validation enforces security and compliance standards. -- **Environment-Specific Configurations**: Supports varying configurations across many teams with their own development, staging, and production environments. -- **Default Value Management**: Provides a mechanism for default settings that can be overridden as needed. - -**Example**: - -```python -class ConfigurationManager: - """ - Manages the complex configuration hierarchy: - 1. Default values - 2. Stack-specific overrides - 3. Environment variables - 4. User-provided configurations - 5. Module-specific settings - """ - - def get_config(self, module_name: str) -> Dict[str, Any]: - # Logic to merge configurations - pass -``` - -### `deployment.py` - -**Purpose**: Orchestrates the deployment process, managing the lifecycle of resources and modules. - -**Key Features**: - -- **Automatic Module Discovery**: Dynamically loads and initializes modules from the `pulumi/modules/` directory. -- **Dependency Resolution**: Ensures resources are created in the correct order based on dependencies. -- **Parallel Deployment Capabilities**: Optimizes deployment time by parallelizing independent resource creations. -- **Comprehensive Error Handling**: Provides robust mechanisms to handle failures and rollbacks. -- **State Management**: Interfaces with Pulumi's state management to track resource states. - -**Example**: - -```python -class DeploymentManager: - """ - Manages the deployment lifecycle: - - Module discovery and loading - - Dependency resolution - - Resource creation ordering - - Error handling and recovery - - State management - """ - - def deploy(self): - # Logic to orchestrate deployment - pass -``` - -### `metadata.py` - -**Purpose**: Handles global metadata and labeling for resources. - -**Key Features**: - -- **Compliance Labels**: Automatically generates and distributes required compliance and audit tags to resources. -- **Git Information Integration**: Incorporates Git metadata (e.g., commit hashes) into resource tags for traceability. -- **Resource Tagging Standards**: Enforces consistent tagging across all modules leaving module maintainers free to implement provider specific tagging implementations. -- **Audit Trail Maintenance**: Facilitates auditing by returning reportable metadata. - -**Example**: - -```python -class MetadataManager: - """ - Manages global metadata: - - Compliance labels - - Git information - - Resource tagging - - Audit trail - """ - - def get_metadata(self) -> Dict[str, str]: - # Returns standardized metadata for resources - pass -``` - -### `resource_helpers.py` - -**Purpose**: Centralizes common resource creation patterns and utilities. - -**Key Features**: - -- **Consistent Metadata Application**: Ensures all resources have the necessary metadata and tags. -- **Standardized Error Handling**: Provides consistent mechanisms for handling resource creation errors. -- **Resource Dependencies**: Simplifies the management of resource dependencies and ordering. -- **Cleanup Procedures**: Implements standardized procedures for resource deletion and cleanup. - -**Example**: - -```python -class ResourceManager: - """ - Provides standardized resource creation patterns: - - Consistent metadata application - - Error handling - - Resource dependencies - - Cleanup procedures - """ - - def create_resource(self, config: Dict[str, Any]) -> pulumi.Resource: - # Logic to create a resource with standardized settings - pass -``` - -### `types.py` - -**Purpose**: Defines shared type definitions and schemas used across modules. - -**Key Features**: - -- **Type Definitions**: Provides base classes and type hints for configurations and resources. -- **Schema Validation**: Ensures that configurations adhere to expected structures. -- **Reusability**: Facilitates consistent typing across different modules. - -**Example**: - -```python -from typing import TypedDict - -class BaseConfig(TypedDict): - """Base configuration structure for all modules.""" - enabled: bool - version: str - parameters: Dict[str, Any] - -class ResourceConfig(TypedDict): - """Standard resource configuration structure.""" - name: str - type: str - metadata: Dict[str, Any] -``` - -### `utils.py` - -**Purpose**: Contains general utility functions that support other components. - -**Key Features**: - -- **Common Utilities**: Functions for tasks like merging configurations, handling strings, etc. -- **Helper Functions**: Small, reusable functions that are used across multiple modules. -- **Performance Optimization**: Includes functions that enhance performance and efficiency. - -**Example**: - -```python -def merge_configurations(*configs: Dict[str, Any]) -> Dict[str, Any]: - """Merges multiple configuration dictionaries into one.""" - # Logic to merge dictionaries - pass -``` - -## Kubernetes Module Centralized Version Control - -**Purpose**: The Kubernetes module centralizes version control to ensure consistency and reliability across all deployments. - -**Reasons for Centralization**: - -- **Consistency**: Guarantees that all modules and environments can opt in to a centralized, common version control system. -- **Simplified Upgrades**: Streamlines the process of maintaining component versions at scale. -- **Compliance and Security**: Ensures that only approved and secure versions are used, adhering to organizational policies. -- **Reduced Conflicts**: Minimizes the risk of version incompatibilities between different modules. - -By centralizing version control in the core module, teams can focus on their specific functionalities without worrying about underlying Kubernetes version discrepancies. - -## Main Entry Point `__main__.py` - -The `__main__.py` file serves as the simple entry point for the Pulumi program. It is designed to be straightforward, allowing different module teams to maintain independent configuration specifications without interference. - -**Key Features**: - -- **Simplified Orchestration**: Delegates the deployment process to the `DeploymentManager` in `deployment.py`. -- **Team Autonomy**: Each team's module can be developed and deployed independently. -- **Preventing Cross-Team Interference**: Changes in one module's deployment do not affect others. - -**Example**: - -```python -import pulumi -from core.deployment import DeploymentManager - -# Initialize the deployment manager -deployment_manager = DeploymentManager() - -# Execute the deployment process -deployment_manager.deploy() -``` - -## When to Use Core Module vs. Module-Specific Code - -### Core Module Code - -**Use the core module when**: - -- The functionality is **used by multiple modules**. -- It **implements organizational standards** or compliance requirements. -- It **provides common resource patterns** that promote consistency. -- You need to **abstract repetitive code** to make it DRY. - -**Example**: - -```python -# Core module - get module configuration -def get_module_configuration(config: Dict[str, Any]) -> Dict[str, Any]: - """Returns the module configuration.""" - pass -``` - -### Module-Specific Code - -**Keep code in individual modules when**: - -- It's **specific to one module's functionality**. -- It implements **module-specific business logic**. -- It handles **module-specific resource types** not used elsewhere. -- It doesn't need to be shared across modules. - -**Example**: - -```python -# AWS module - reusable but only within AWS module resource patterns -def create_compliant_s3_bucket(name: str, config: Dict[str, Any]) -> s3.Bucket: - """Creates an S3 bucket with standard compliance controls.""" - # Logic to create the bucket - pass -``` - -## Best Practices - -- **Maintain Type Safety**: Use type hints and `TypedDict` to prevent runtime errors. -- **Follow the DRY Principle**: Abstract common code into the core module to avoid duplication. -- **Isolate Configurations**: Keep module configurations independent to prevent conflicts. -- **Document Thoroughly**: Provide clear docstrings and comments for all functions and classes. -- **Implement Comprehensive Error Handling**: Use custom exceptions and handle errors gracefully. -- **Use Centralized Metadata**: Apply consistent tagging and labeling using the `metadata.py` utilities. -- **Adhere to Compliance Standards**: Leverage the core module's compliance features to meet organizational policies. - -## Development Guidelines - -- **Adding New Core Features**: - - - **Evaluate the Need**: - - Is the functionality used by multiple modules? - - Does it align with organizational standards? - - Should it be centralized for consistency? - - - **Design**: - - Create clear type definitions. - - Plan for error handling and edge cases. - - Consider backward compatibility. - - - **Implementation**: - - Write type-safe, clean code. - - Add comprehensive unit tests. - - Document the new feature thoroughly. - -- **Testing**: - - - **Unit Tests**: Test individual components for correctness and type safety. - - **Integration Tests**: Verify interactions between components and modules. - - **Compliance Tests**: Ensure new features meet security and compliance requirements. - -- **Contributing**: - - - **Propose Changes**: Discuss your ideas with maintainers before implementation. - - **Follow Code Standards**: Adhere to the project's coding guidelines. - - **Submit Pull Requests**: Provide detailed descriptions and await code reviews. diff --git a/docs/developer_guide/pulumi_python.md b/docs/developer_guide/pulumi_python.md new file mode 100644 index 0000000..a315236 --- /dev/null +++ b/docs/developer_guide/pulumi_python.md @@ -0,0 +1,601 @@ +# Pulumi Python Development Guide: Advanced Techniques and Best Practices + +This guide provides an in-depth exploration of developing Infrastructure as Code (IaC) projects using Pulumi with Python. It focuses on advanced coding techniques, strategic decision-making, style guidelines, and unique considerations that arise when working with state-driven IaC tools. + +--- + +## Table of Contents + +1. [Introduction](#introduction) +2. [Getting Started with Pulumi and Python](#getting-started-with-pulumi-and-python) + - [Project Setup](#project-setup) + - [Dependency Management with Poetry](#dependency-management-with-poetry) + - [Configuring Pulumi to Use Poetry](#configuring-pulumi-to-use-poetry) +3. [Python Coding Standards for Pulumi IaC](#python-coding-standards-for-pulumi-iac) + - [Code Style and Conventions](#code-style-and-conventions) + - [Type Annotations and Type Safety](#type-annotations-and-type-safety) + - [Error Handling and Logging](#error-handling-and-logging) +4. [Strategic Coding Techniques and Style Decisions](#strategic-coding-techniques-and-style-decisions) + - [When to Use Classes vs. Functions](#when-to-use-classes-vs-functions) + - [Using `TypedDict` and Pydantic Models](#using-typeddict-and-pydantic-models) + - [Modular Code Organization](#modular-code-organization) + - [Clean and Readable Code Practices](#clean-and-readable-code-practices) +5. [Unique IaC Development Considerations](#unique-iac-development-considerations) + - [State-Based Orchestration Principles](#state-based-orchestration-principles) + - [Resource Lifecycle Management](#resource-lifecycle-management) + - [Idempotency and State Management](#idempotency-and-state-management) +6. [Advanced Pulumi Patterns in Python](#advanced-pulumi-patterns-in-python) + - [Dynamic Resource Management](#dynamic-resource-management) + - [Handling Dependencies](#handling-dependencies) + - [Resource Options and Customizations](#resource-options-and-customizations) +7. [Configuration Management](#configuration-management) + - [Using Pulumi Config and Secrets](#using-pulumi-config-and-secrets) + - [Managing Environments and Stacks](#managing-environments-and-stacks) + - [Best Practices for Configuration Files](#best-practices-for-configuration-files) +8. [Testing and Validation](#testing-and-validation) + - [Unit Testing with Pulumi and Python](#unit-testing-with-pulumi-and-python) + - [Integration Testing Strategies](#integration-testing-strategies) + - [Enforcing Type Checking with Pyright](#enforcing-type-checking-with-pyright) +9. [Style Guide for Pulumi Python Projects](#style-guide-for-pulumi-python-projects) + - [Naming Conventions](#naming-conventions) + - [Formatting and Linting Tools](#formatting-and-linting-tools) + - [Docstrings and Documentation](#docstrings-and-documentation) +10. [Best Practices](#best-practices) + - [DRY Principle in IaC](#dry-principle-in-iac) + - [Security Considerations](#security-considerations) + - [Code Reusability and Modularity](#code-reusability-and-modularity) +11. [Conclusion](#conclusion) +12. [References and Further Reading](#references-and-further-reading) + +--- + +## Introduction + +Pulumi allows developers to define and manage cloud infrastructure using familiar programming languages, including Python. This guide dives deep into advanced Python coding practices tailored for Pulumi IaC projects. It aims to help developers make strategic coding decisions, adhere to style guidelines, and understand unique considerations when working with state-driven IaC tools. + +--- + +## Getting Started with Pulumi and Python + +### Project Setup + +1. **Install Pulumi CLI**: [Download Pulumi](https://www.pulumi.com/docs/get-started/install/). +2. **Install Python**: Ensure you have Python 3.8 or higher. +3. **Install Poetry**: [Install Poetry](https://python-poetry.org/docs/#installation). +4. **Initialize a New Pulumi Project**: + + ```bash + pulumi new aws-python + ``` + +### Dependency Management with Poetry + +Poetry simplifies dependency management and virtual environment creation. + +1. **Initialize Poetry**: + + ```bash + poetry init + ``` + + Follow the prompts to set up your `pyproject.toml`. + +2. **Add Dependencies**: + + ```bash + poetry add pulumi pulumi-aws + ``` + +3. **Activate the Virtual Environment**: + + ```bash + poetry shell + ``` + +### Configuring Pulumi to Use Poetry + +Ensure Pulumi uses the Poetry-managed environment: + +1. **Update `Pulumi.yaml`**: + + ```yaml + name: your-pulumi-project + runtime: + name: python + options: + virtualenv: poetry + ``` + +2. **Install Pulumi Dependencies**: + + ```bash + pulumi install + ``` + +--- + +## Python Coding Standards for Pulumi IaC + +### Code Style and Conventions + +- **PEP 8 Compliance**: Follow the [PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/). +- **Naming Conventions**: + - Variables and functions: `snake_case` + - Classes and exceptions: `PascalCase` + - Constants: `UPPER_SNAKE_CASE` + +### Type Annotations and Type Safety + +- **Use Type Hints**: Provide type annotations for all functions and methods. + + ```python + def create_bucket(name: str, versioning: bool) -> s3.Bucket: + ... + ``` + +- **Enforce Type Checking**: Use tools like Pyright for static type analysis. + +### Error Handling and Logging + +- **Explicit Exceptions**: Raise specific exceptions with clear messages. + + ```python + if not valid: + raise ValueError("Invalid configuration parameter") + ``` + +- **Use Pulumi Logging**: + + ```python + from pulumi import log + + log.info("Resource created successfully") + log.error("Failed to create resource") + ``` + +--- + +## Strategic Coding Techniques and Style Decisions + +### When to Use Classes vs. Functions + +- **Classes**: Use when you need to encapsulate state and behavior. + + ```python + class S3BucketManager: + def __init__(self, bucket_name: str): + self.bucket_name = bucket_name + + def create_bucket(self) -> s3.Bucket: + return s3.Bucket(self.bucket_name) + ``` + +- **Functions**: Use for stateless operations or simple tasks. + + ```python + def create_bucket(name: str) -> s3.Bucket: + return s3.Bucket(name) + ``` + +### Using `TypedDict` and Pydantic Models + +#### `TypedDict` for Configurations + +```python +from typing import TypedDict + +class BucketConfig(TypedDict): + name: str + versioning: bool + tags: dict +``` + +#### Pydantic Models for Validation + +```python +from pydantic import BaseModel, Field + +class BucketConfigModel(BaseModel): + name: str = Field(..., description="Name of the S3 bucket") + versioning: bool = Field(default=False, description="Enable versioning") + tags: dict = Field(default_factory=dict, description="Bucket tags") +``` + +### Modular Code Organization + +- **Packages and Modules**: Organize code into logical packages. + + ``` + my_project/ + ├── __init__.py + ├── networking/ + │ ├── __init__.py + │ └── vpc.py + ├── storage/ + │ ├── __init__.py + │ └── s3.py + └── main.py + ``` + +- **Reusability**: Encapsulate functionality into reusable components. + +### Clean and Readable Code Practices + +- **Use `*args` and `**kwargs`**: For flexible function arguments. + + ```python + from typing import Any, Dict + import pulumi_aws as aws + + def create_security_group(name: str, *ingress_rules: Dict[str, Any], **kwargs): + return aws.ec2.SecurityGroup( + resource_name=name, + ingress=list(ingress_rules), + **kwargs + ) + + # Usage + sg = create_security_group( + "web-sg", + {"protocol": "tcp", "from_port": 80, "to_port": 80, "cidr_blocks": ["0.0.0.0/0"]}, + {"protocol": "tcp", "from_port": 443, "to_port": 443, "cidr_blocks": ["0.0.0.0/0"]}, + tags={"Environment": "Production"} + ) + ``` + + **Benefits**: + + - Allows passing any number of ingress rules without modifying the function signature. + - Enables flexible resource creation with additional arguments via `**kwargs`. + +- **List Comprehensions**: Simplify resource creation. + + ```python + import pulumi_aws as aws + + bucket_names = ["logs", "data", "backup"] + + buckets = [ + aws.s3.Bucket( + f"{name}-bucket", + bucket=f"mycompany-{name}-bucket" + ) for name in bucket_names + ] + ``` + + **Benefits**: + + - Reduces code verbosity by eliminating explicit loops. + - Enhances readability by keeping related code in a single expression. + +- **Utilize `zip()` for Parallel Resource Configuration**: + + ```python + import pulumi_aws as aws + + instance_types = ["t2.micro", "t2.small", "t2.medium"] + ami_ids = ["ami-123", "ami-456", "ami-789"] + names = ["web-server", "app-server", "db-server"] + + instances = [ + aws.ec2.Instance( + f"{name}-instance", + instance_type=instance_type, + ami=ami_id, + tags={"Name": name} + ) + for instance_type, ami_id, name in zip(instance_types, ami_ids, names) + ] + ``` + + **Benefits**: + + - Simplifies the creation of resources with paired configurations. + - Ensures that corresponding elements from multiple lists are used together. + + **Note**: This pattern is useful when configurations are aligned and should be used appropriately. + +- **Merging Dictionaries for Resource Arguments**: + + ```python + default_tags = {"Project": "MyProject", "ManagedBy": "Pulumi"} + environment_tags = {"Environment": "Production"} + + all_tags = {**default_tags, **environment_tags} + + bucket = aws.s3.Bucket( + "my-bucket", + tags=all_tags + ) + ``` + + **Benefits**: + + - Avoids repetitive code by reusing common configurations. + - Enhances maintainability by centralizing default settings. + +- **Chained Comparisons**: + + ```python + if 0 < value < 10: + ... + ``` + +- **Ternary Operators**: + + ```python + environment = pulumi.get_stack() # e.g., 'dev' or 'prod' + + db_name = "mydb-prod" if environment == "prod" else "mydb-dev" + + rds_instance = aws.rds.Instance( + "db-instance", + instance_class="db.t2.micro", + allocated_storage=20, + name=db_name + ) + ``` + + **Benefits**: + + - Reduces code lines for simple conditional assignments. + - Enhances readability by keeping related logic together. + +--- + +## Unique IaC Development Considerations + +### State-Based Orchestration Principles + +- **Desired State Configuration**: Define what the infrastructure should look like, not how to achieve it. +- **Idempotency**: Repeated executions should produce the same result. +- **State Synchronization**: Pulumi tracks resource states to determine changes. + +### Resource Lifecycle Management + +- **Creation**: Define resources in code; Pulumi provisions them. +- **Update**: Modify the code; Pulumi updates the resources accordingly. +- **Deletion**: Remove resource definitions; Pulumi destroys them. + +### Idempotency and State Management + +- **Avoid Mutable Global State**: Use function parameters and return values. +- **Explicit State Definitions**: Clearly define resource states in code. +- **Handle Dependencies**: Use Pulumi's dependency management to ensure correct resource creation order. + +--- + +## Advanced Pulumi Patterns in Python + +### Dynamic Resource Management + +- **Dynamic Providers**: Create resources that are not natively supported. + + ```python + from pulumi.dynamic import Resource, ResourceProvider, CreateResult + + class MyResourceProvider(ResourceProvider): + def create(self, props): + # Custom resource creation logic + return CreateResult(id_, outs) + + class MyResource(Resource): + def __init__(self, name, props, opts=None): + super().__init__(MyResourceProvider(), name, props, opts) + ``` + +### Handling Dependencies + +- **Explicit Dependencies**: + + ```python + bucket = s3.Bucket("my-bucket") + bucket_object = s3.BucketObject( + "object", + bucket=bucket.id, + opts=pulumi.ResourceOptions(depends_on=[bucket]) + ) + ``` + +### Resource Options and Customizations + +- **Resource Options**: Customize resource behavior. + + ```python + s3.Bucket( + "my-bucket", + opts=pulumi.ResourceOptions(protect=True) + ) + ``` + +- **Aliases for Renamed Resources**: Handle resource renaming without recreation. + + ```python + old_bucket = s3.Bucket("old-name") + new_bucket = s3.Bucket( + "new-name", + opts=pulumi.ResourceOptions(aliases=[pulumi.Alias(name="old-name")]) + ) + ``` + +--- + +## Configuration Management + +### Using Pulumi Config and Secrets + +- **Access Configuration Values**: + + ```python + import pulumi + + config = pulumi.Config() + db_password = config.require_secret("db_password") + ``` + +- **Set Configuration Values**: + + ```bash + pulumi config set aws:region us-west-2 + pulumi config set --secret db_password yourpassword + ``` + +### Managing Environments and Stacks + +- **Stacks**: Isolated instances of your Pulumi program (e.g., dev, staging, prod). +- **Stack-Specific Configurations**: + + ```bash + pulumi stack init dev + pulumi config set aws:region us-west-2 --stack dev + ``` + +### Best Practices for Configuration Files + +- **Avoid Hardcoding Values**: Use configuration files and Pulumi Config. +- **Use Secrets Management**: Store sensitive data securely. +- **Environment Variables**: Use cautiously; prefer Pulumi Config for consistency. + +--- + +## Testing and Validation + +### Unit Testing with Pulumi and Python + +- **Mocking Resources**: Use Pulumi's testing framework to mock resource providers. + + ```python + import pulumi + from pulumi.runtime import mocks + + class MyMocks(mocks.Mocks): + def new_resource(self, type_, name, inputs, provider, id_): + return [name + "_id", inputs] + + def call(self, token, args, provider): + return {} + + pulumi.runtime.set_mocks(MyMocks()) + ``` + +### Integration Testing Strategies + +- **End-to-End Tests**: Deploy to a test environment and validate resource behavior. +- **Test Infrastructure Code Separately**: Keep tests isolated from production resources. + +### Enforcing Type Checking with Pyright + +- **Install Pyright**: + + ```bash + poetry add --dev pyright + ``` + +- **Configure Pyright**: + + ```json + { + "include": ["**/*.py"], + "exclude": ["**/__pycache__/**"], + "reportMissingImports": true, + "pythonVersion": "3.8", + "typeCheckingMode": "strict" + } + ``` + +- **Run Pyright**: + + ```bash + poetry run pyright + ``` + +--- + +## Style Guide for Pulumi Python Projects + +### Naming Conventions + +- **Resources**: Use descriptive names reflecting their purpose. + + ```python + bucket = s3.Bucket("user-uploads-bucket") + ``` + +- **Variables and Functions**: `snake_case` +- **Classes and Exceptions**: `PascalCase` + +### Formatting and Linting Tools + +- **Black**: Code formatter. + + ```bash + poetry add --dev black + black . + ``` + +- **Flake8**: Linting tool. + + ```bash + poetry add --dev flake8 + flake8 . + ``` + +### Docstrings and Documentation + +- **Use Docstrings**: Document modules, classes, and functions. + + ```python + def create_bucket(name: str) -> s3.Bucket: + """ + Create an S3 bucket with the given name. + + Args: + name: The name of the bucket. + + Returns: + The created S3 bucket resource. + """ + ... + ``` + +--- + +## Best Practices + +### DRY Principle in IaC + +- **Avoid Repetition**: Use functions and classes to encapsulate reusable code. +- **Shared Modules**: Create modules for common resources or patterns. + +### Security Considerations + +- **Secrets Management**: Use Pulumi's secrets to store sensitive data. +- **Least Privilege**: Grant minimal necessary permissions to resources. +- **Regular Audits**: Review configurations for security compliance. + +### Code Reusability and Modularity + +- **Parameterization**: Make resources configurable via inputs. +- **Version Control**: Keep IaC code in a VCS like Git. +- **Module Packaging**: Distribute reusable components as Python packages. + +--- + +## Conclusion + +Developing IaC projects with Pulumi and Python offers flexibility and power, enabling developers to leverage familiar programming constructs while managing complex infrastructure. By adhering to strategic coding techniques, style guidelines, and understanding unique IaC considerations, you can create maintainable, scalable, and robust infrastructure codebases. + +--- + +## References and Further Reading + +- **Pulumi Python Documentation**: [Pulumi Python Docs](https://www.pulumi.com/docs/intro/languages/python/) +- **PEP 8 Style Guide**: [PEP 8](https://www.python.org/dev/peps/pep-0008/) +- **Pyright Type Checker**: [Pyright GitHub](https://github.com/microsoft/pyright) +- **Poetry Documentation**: [Poetry Docs](https://python-poetry.org/docs/) +- **Pydantic Models**: [Pydantic Documentation](https://pydantic-docs.helpmanual.io/) +- **TypedDict Documentation**: [PEP 589 - TypedDict](https://www.python.org/dev/peps/pep-0589/) + +--- + +**Note**: This guide is intended to supplement the [Konductor Developer Guide](./README.md) by providing deeper technical insights and best practices specific to Pulumi Python development. diff --git a/docs/developer_guide/modules/aws/README.md b/docs/developer_guide/roadmaps/aws/README.md similarity index 100% rename from docs/developer_guide/modules/aws/README.md rename to docs/developer_guide/roadmaps/aws/README.md diff --git a/docs/developer_guide/modules/aws/developer_guide.md b/docs/developer_guide/roadmaps/aws/developer_guide.md similarity index 100% rename from docs/developer_guide/modules/aws/developer_guide.md rename to docs/developer_guide/roadmaps/aws/developer_guide.md diff --git a/docs/developer_guide/modules/aws/eks_donor_template.md b/docs/developer_guide/roadmaps/aws/eks_donor_template.md similarity index 100% rename from docs/developer_guide/modules/aws/eks_donor_template.md rename to docs/developer_guide/roadmaps/aws/eks_donor_template.md diff --git a/docs/developer_guide/modules/aws/eks_opentelemetry_docs.md b/docs/developer_guide/roadmaps/aws/eks_opentelemetry_docs.md similarity index 100% rename from docs/developer_guide/modules/aws/eks_opentelemetry_docs.md rename to docs/developer_guide/roadmaps/aws/eks_opentelemetry_docs.md diff --git a/docs/developer_guide/modules/aws/implementation_roadmap.md b/docs/developer_guide/roadmaps/aws/implementation_roadmap.md similarity index 100% rename from docs/developer_guide/modules/aws/implementation_roadmap.md rename to docs/developer_guide/roadmaps/aws/implementation_roadmap.md diff --git a/docs/getting_started.md b/docs/getting_started.md deleted file mode 100644 index 6bd1067..0000000 --- a/docs/getting_started.md +++ /dev/null @@ -1,165 +0,0 @@ -# Getting Started with Konductor - -Welcome to Konductor! This guide will help you quickly set up and begin using the Konductor Infrastructure as Code (IaC) platform. Whether you're new to DevOps or an experienced platform engineer, this guide provides everything you need to get started. - -## Table of Contents - -1. [Prerequisites](#prerequisites) -2. [Quick Setup](#quick-setup) -3. [Initial Configuration](#initial-configuration) -4. [First Deployment](#first-deployment) -5. [Next Steps](#next-steps) -6. [Troubleshooting](#troubleshooting) -7. [Getting Help](#getting-help) - -## Prerequisites - -Before you begin, ensure you have the following installed: - -- **Python**: Version 3.8 or higher -- **Poetry**: For dependency management ([Install Poetry](https://python-poetry.org/docs/#installation)) -- **Pulumi CLI**: For infrastructure management ([Install Pulumi](https://www.pulumi.com/docs/get-started/install/)) -- **Git**: For version control -- **AWS CLI** (optional): If working with AWS resources -- **kubectl** (optional): If working with Kubernetes resources - -> **Note**: All dependencies are automatically supplied if you use the provided Dev Container with VS Code. - -## Quick Setup - -### 1. Create Your Project - -Choose one of these methods to create your project: - -#### Option A: Use GitHub Template (Recommended) - -1. Visit the [Konductor Template Repository](https://github.com/containercraft/konductor). -2. Click "Use this template." -3. Fill in your repository details. -4. Clone your new repository: - ```bash - git clone https://github.com/your-username/your-project-name.git - cd your-project-name - ``` - -#### Option B: Clone Directly - -```bash -git clone https://github.com/containercraft/konductor.git -cd konductor -``` - -### 2. Set Up Development Environment - -#### Using Dev Container (Recommended) - -1. Install [VS Code](https://code.visualstudio.com/) and the [Remote - Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). -2. Open the project in VS Code. -3. When prompted, click "Reopen in Container." - -#### Manual Setup - -1. Install dependencies with Poetry: - ```bash - poetry install - ``` - -2. Activate the virtual environment: - ```bash - poetry shell - ``` - -## Initial Configuration - -### 1. Configure Pulumi - -1. Log in to Pulumi: - ```bash - pulumi login - ``` - -2. Create a new stack: - ```bash - pulumi stack init dev - ``` - -3. Configure your cloud provider (example for AWS): - ```bash - pulumi config set aws:region us-west-2 - ``` - -### 2. Set Up Cloud Provider Credentials - -#### For AWS: - -```bash -aws configure -``` - -Follow the prompts to enter your AWS credentials. - -> **Security Note**: Always follow your organization's security policies when handling credentials. - -## First Deployment - -1. **Preview Changes**: - ```bash - pulumi preview - ``` - -2. **Deploy Infrastructure**: - ```bash - pulumi up - ``` - -3. **Verify Deployment**: - ```bash - pulumi stack output - ``` - -## Next Steps - -After completing your initial setup, explore these resources: - -1. **[User Guide](./user_guide/README.md)** - - Learn about Konductor's features and capabilities. - - Understand best practices for using the platform. - -2. **[Module Documentation](./modules/README.md)** - - Explore available modules. - - Learn how to use specific modules like AWS or cert-manager. - -3. **[Developer Guide](./developer_guide/README.md)** - - Contribute to Konductor. - - Create custom modules. - -## Troubleshooting - -Common issues and their solutions: - -### Poetry Installation Issues - -**Problem**: Poetry installation fails. -**Solution**: Try installing with pip: - - ```bash - pip install --user poetry - ``` - -### Pulumi Login Issues - -**Problem**: Cannot log in to Pulumi. -**Solution**: Ensure you have created an account at [app.pulumi.com](https://app.pulumi.com) and create a new Personal Access Token (PAT). - -For more troubleshooting help, see our [FAQ and Troubleshooting Guide](./user_guide/faq_and_troubleshooting.md). - -## Getting Help - -- **Documentation**: Browse our [comprehensive documentation](./README.md). -- **Community**: Join our [Discord community](https://discord.gg/Jb5jgDCksX). -- **Issues**: Report problems on our [GitHub Issues](https://github.com/containercraft/konductor/issues). -- **Discussions**: Start a discussion in our [GitHub Discussions](https://github.com/containercraft/konductor/discussions). - ---- - -**Next**: Explore the [User Guide](./user_guide/README.md) to learn more about using Konductor effectively. diff --git a/docs/infrastructure_as_code.md b/docs/infrastructure_as_code.md deleted file mode 100644 index e590bbd..0000000 --- a/docs/infrastructure_as_code.md +++ /dev/null @@ -1,131 +0,0 @@ -# Python Developer Guidelines for Infrastructure as Code (IaC) - -## Table of Contents - -1. [Introduction](#introduction) -2. [Understanding Infrastructure as Code](#understanding-infrastructure-as-code) -3. [State-Based Orchestration Principles](#state-based-orchestration-principles) -4. [Python Coding Patterns for IaC](#python-coding-patterns-for-iac) -5. [Modular Design and Reusability](#modular-design-and-reusability) -6. [State Management and Idempotency](#state-management-and-idempotency) -7. [Resource Lifecycle Management](#resource-lifecycle-management) -8. [Best Practices](#best-practices) -9. [Practical Examples](#practical-examples) -10. [Conclusion](#conclusion) -11. [Further Reading](#further-reading) - -## Introduction - -This document provides comprehensive guidelines for Python developers aiming to master Infrastructure as Code (IaC) principles, focusing on coding patterns and state-based orchestration. It caters to: - -- **Junior Developers**: Proficient in Python but new to IaC and modular design. -- **Senior Developers**: Experienced in cloud automation but unfamiliar with declarative resource management in IaC. -- **Principal Developers**: Skilled in advanced Python architectures but new to IaC in general-purpose languages. -- **AI Coding Assistants**: Seeking high-quality, detailed documentation with maximal semantic density. - -## Understanding Infrastructure as Code - -**Infrastructure as Code (IaC)** is the practice of provisioning and managing computing infrastructure through machine-readable definition files, rather than manual hardware configuration or interactive configuration tools. - -- **Declarative vs. Imperative**: Declarative IaC specifies *what* the desired state is, while imperative IaC specifies *how* to achieve it. -- **Benefits of IaC**: - - **Reproducibility**: Ensures consistent environments across deployments. - - **Version Control**: Infrastructure definitions can be versioned like application code. - - **Automation**: Reduces manual errors and speeds up deployment. - -## State-Based Orchestration Principles - -State-based orchestration manages resources by comparing the desired state (as defined in code) with the current state of the infrastructure. - -- **Desired State Configuration**: Define the intended state without specifying the steps to achieve it. -- **Idempotency**: Applying the same configuration multiple times yields the same result. -- **State Synchronization**: Tools track resource states to determine necessary actions. - -## Python Coding Patterns for IaC - -Using Python for IaC combines the versatility of a general-purpose language with infrastructure management. - -- **Advantages**: - - **Expressiveness**: Leverage Python's syntax and libraries. - - **Modularity**: Organize code into reusable modules. - - **Integration**: Utilize existing Python ecosystems and tools. - -### Key Patterns: - -- **Abstraction**: Encapsulate infrastructure components into classes and functions. -- **Encapsulation**: Hide implementation details, exposing only necessary interfaces. -- **Composition**: Build complex systems by combining simpler components. - -## Modular Design and Reusability - -Modular design promotes maintainability and scalability. - -- **Modules**: Self-contained units of code representing infrastructure components. -- **Reusability**: Write modules once and use them across different projects. -- **Encapsulation**: Prevent external code from depending on internal module details. - -### Implementation Strategies: - -- **Use Classes and Functions**: Encapsulate logic for resource creation. -- **Parameterization**: Allow customization through input parameters. -- **Naming Conventions**: Adopt consistent naming for clarity. - -## State Management and Idempotency - -Managing state is crucial for predictable infrastructure behavior. - -- **State Files**: Store metadata about deployed resources. -- **Idempotent Operations**: Ensure that repeated executions don't produce unintended changes. -- **Change Detection**: Compare desired and actual states to determine necessary updates. - -### Best Practices: - -- **Avoid Mutable Global State**: Use function parameters and return values. -- **Explicit State Definitions**: Clearly define resource states in code. -- **State Isolation**: Separate state management from business logic. - -## Resource Lifecycle Management - -Manage resources by defining or removing their configurations. - -- **Creation**: Define resources in code; the orchestration tool provisions them. -- **Deletion**: Remove resource definitions; the tool destroys them automatically. -- **No Explicit Destroy Logic**: Rely on the orchestration tool's state management. - -### Benefits: - -- **Simplicity**: Focus on desired state without handling low-level details. -- **Consistency**: Reduce errors by automating resource cleanup. -- **Efficiency**: Save time by not writing repetitive destroy code. - -## Best Practices - -- **Version Control Everything**: Keep all code, including infrastructure definitions, in a VCS. -- **Write Tests**: Implement unit and integration tests for your IaC code. -- **Documentation**: Maintain clear and concise documentation for modules and functions. -- **Security**: Protect sensitive data using secrets management tools. -- **Code Reviews**: Regularly review code to ensure adherence to standards. - -## Practical Examples - -### Example 1: Creating an AWS S3 Bucket with Pulumi - -```python -import pulumi -from pulumi_aws import s3 - -# Create an AWS S3 bucket -bucket = s3.Bucket('my-bucket') - -# Export the bucket name -pulumi.export('bucket_name', bucket.id) -``` - -### Example 2: Deleting a Resource by Removing Its Definition - -- **Before Deletion**: Resource is defined in code. -- **After Deletion**: Remove the resource code; run the orchestration tool to update the infrastructure. - -## Conclusion - -Mastering IaC with Python involves understanding state-based orchestration, adopting proper coding patterns, and emphasizing modularity and idempotency. By defining desired states and relying on orchestration tools, developers can efficiently manage infrastructure lifecycles without explicit destroy logic. diff --git a/docs/modules/core/README.md b/docs/modules/core/README.md new file mode 100644 index 0000000..cf0bb6e --- /dev/null +++ b/docs/modules/core/README.md @@ -0,0 +1,466 @@ +# Konductor Core Module Documentation + +## Introduction + +The **Konductor Core Module** is the foundational framework for the Konductor Infrastructure as Code (IaC) platform built with Pulumi and Python. It provides a robust, type-safe, and maintainable architecture that enables development teams to work independently while ensuring consistency, compliance, and best practices across all infrastructure deployments. + +This documentation is intended for developers of all levels, DevOps practitioners, module maintainers, and contributors. It serves as a comprehensive guide to understanding, utilizing, and extending the core module effectively and confidently. + +--- + +## Table of Contents + +1. [Architecture Overview](#architecture-overview) + - [Directory Structure](#directory-structure) + - [Design Philosophy](#design-philosophy) +2. [Core Module Components](#core-module-components) + - [`__init__.py`](#__init__py) + - [`config.py`](#configpy) + - [`deployment.py`](#deploymentpy) + - [`metadata.py`](#metadatapy) + - [`resource_helpers.py`](#resource_helperspy) + - [`types.py`](#typespy) + - [`utils.py`](#utilspy) +3. [Configuration Management](#configuration-management) + - [Complexity Rationale](#complexity-rationale) + - [Configuration Layers](#configuration-layers) +4. [Deployment Orchestration](#deployment-orchestration) +5. [Metadata and Tagging](#metadata-and-tagging) +6. [Resource Management](#resource-management) +7. [Type Definitions and Schemas](#type-definitions-and-schemas) +8. [Utility Functions](#utility-functions) +9. [Usage Guidelines](#usage-guidelines) + - [When to Use the Core Module](#when-to-use-the-core-module) + - [Extending the Core Module](#extending-the-core-module) +10. [Best Practices](#best-practices) +11. [Development Guidelines](#development-guidelines) +12. [Contribution Guidelines](#contribution-guidelines) +13. [Conclusion](#conclusion) +14. [Related Documentation](#related-documentation) + +--- + +## Architecture Overview + +### Directory Structure + +The core module is organized to promote clarity, maintainability, and ease of use: + +``` +core/ +├── __init__.py # Module initialization and public API exports +├── config.py # Advanced configuration management system +├── deployment.py # Deployment orchestration and lifecycle management +├── metadata.py # Global metadata and tagging utilities +├── resource_helpers.py # Common resource creation and management functions +├── types.py # Shared type definitions and schemas +└── utils.py # General utility functions +``` + +### Design Philosophy + +The core module is designed with the following principles: + +1. **Separation of Concerns**: Each component addresses a specific aspect of the infrastructure management process. +2. **Type Safety**: Comprehensive use of type annotations and `TypedDict` to ensure reliability and prevent runtime errors. +3. **Configuration Isolation**: Enables modules to define and manage their own configurations independently. +4. **Resource Abstraction**: Provides reusable patterns for resource creation to promote consistency. +5. **Compliance Integration**: Incorporates security and compliance requirements seamlessly. +6. **Team Autonomy**: Allows teams to work independently without impacting other modules or the core functionality. + +--- + +## Core Module Components + +### `__init__.py` + +**Purpose**: Initializes the core module and exposes its public API. + +**Key Responsibilities**: + +- Import and expose essential classes and functions. +- Define `__all__` to manage the public interface. +- Provide module-level docstrings that describe the module's purpose and usage. + +**Example**: + +```python +""" +Core module initialization and public API exposure. +""" + +from .config import ConfigurationManager +from .deployment import DeploymentManager +from .metadata import MetadataManager +from .resource_helpers import ResourceManager +from .types import BaseConfig, ResourceConfig +from .utils import merge_configurations + +__all__ = [ + "ConfigurationManager", + "DeploymentManager", + "MetadataManager", + "ResourceManager", + "BaseConfig", + "ResourceConfig", + "merge_configurations", +] +``` + +### `config.py` + +**Purpose**: Manages complex configuration hierarchies across the platform. + +**Key Responsibilities**: + +- Merge configurations from various sources (defaults, environment variables, user inputs). +- Validate configurations against predefined schemas. +- Provide configuration isolation for individual modules. +- Support environment-specific settings. + +**Example**: + +```python +from typing import Dict, Any + +class ConfigurationManager: + """ + Manages layered configurations for modules. + + Configuration precedence: + 1. Global defaults + 2. Environment variables + 3. Stack-specific overrides + 4. User-provided configurations + 5. Module-specific settings + """ + + def __init__(self): + self.global_defaults = self.load_global_defaults() + + def get_config(self, module_name: str) -> Dict[str, Any]: + """Retrieve the merged configuration for a module.""" + module_defaults = self.load_module_defaults(module_name) + user_config = self.load_user_config(module_name) + return self.merge_configurations( + self.global_defaults, + module_defaults, + user_config + ) +``` + +### `deployment.py` + +**Purpose**: Orchestrates the deployment process and manages resource lifecycles. + +**Key Responsibilities**: + +- Discover and load modules dynamically. +- Resolve resource dependencies and creation order. +- Handle errors and implement rollback mechanisms. +- Interface with Pulumi's state management. + +**Example**: + +```python +from typing import List + +class DeploymentManager: + """ + Orchestrates module deployments. + + Responsibilities: + - Module discovery and initialization + - Dependency resolution + - Resource lifecycle management + - Error handling and rollback + - State synchronization with Pulumi + """ + + def deploy(self, modules: List[str]) -> None: + """Deploy specified modules.""" + for module_name in modules: + self.initialize_module(module_name) + self.resolve_dependencies() + self.execute_deployment() +``` + +### `metadata.py` + +**Purpose**: Manages global metadata and resource tagging. + +**Key Responsibilities**: + +- Generate and apply standardized tags for compliance. +- Integrate Git metadata (e.g., commit hashes) for traceability. +- Provide utilities for consistent resource metadata application. + +**Example**: + +```python +from typing import Dict + +class MetadataManager: + """ + Handles global metadata and tagging for resources. + + - Generates compliance tags + - Integrates version control metadata + - Provides methods to apply metadata to resources + """ + + def get_standard_tags(self) -> Dict[str, str]: + """Return a dictionary of standard tags to apply to resources.""" + return { + "Project": "Konductor", + "ManagedBy": "Pulumi", + "GitCommit": self.get_git_commit_hash(), + } + + def apply_tags(self, resource, tags: Dict[str, str]) -> None: + """Apply tags to a given resource.""" + resource.tags.apply(lambda existing_tags: {**existing_tags, **tags}) +``` + +### `resource_helpers.py` + +**Purpose**: Provides common patterns and utilities for resource creation. + +**Key Responsibilities**: + +- Simplify resource creation with standardized methods. +- Ensure consistent application of metadata and compliance controls. +- Manage resource dependencies and error handling. + +**Example**: + +```python +from typing import Dict, Any +import pulumi + +class ResourceManager: + """ + Offers helper methods for resource creation. + + - Standardizes resource instantiation + - Applies metadata and compliance tags + - Manages dependencies and error handling + """ + + def create_resource( + self, + resource_type: Any, + name: str, + args: Dict[str, Any], + opts: pulumi.ResourceOptions = None + ) -> pulumi.Resource: + """Create a resource with standardized settings.""" + metadata_manager = MetadataManager() + standard_tags = metadata_manager.get_standard_tags() + args['tags'] = {**args.get('tags', {}), **standard_tags} + return resource_type(name, **args, opts=opts) +``` + +### `types.py` + +**Purpose**: Defines shared type definitions and data schemas. + +**Key Responsibilities**: + +- Provide base configuration types using `TypedDict` or Pydantic models. +- Ensure consistent data structures across modules. +- Facilitate type checking and validation. + +**Example**: + +```python +from typing import TypedDict, Optional, Dict, Any + +class BaseConfig(TypedDict): + """Base configuration structure for modules.""" + enabled: bool + version: Optional[str] + parameters: Dict[str, Any] +``` + +### `utils.py` + +**Purpose**: Contains general utility functions used across the core module. + +**Key Responsibilities**: + +- Provide common utility functions (e.g., configuration merging, string manipulation). +- Enhance performance and code reusability. +- Support other core components with shared functionalities. + +**Example**: + +```python +from typing import Dict, Any + +def merge_configurations(*configs: Dict[str, Any]) -> Dict[str, Any]: + """Merge multiple configuration dictionaries into one.""" + result = {} + for config in configs: + result.update(config) + return result +``` + +--- + +## Configuration Management + +### Complexity Rationale + +The configuration system is intentionally designed to handle complex scenarios to support: + +- **Team Autonomy**: Allowing different teams to define their own configurations without impacting others. +- **Environment Variability**: Managing configurations across multiple environments (development, staging, production). +- **Compliance Requirements**: Enforcing organizational policies and standards centrally. +- **Scalability**: Accommodating a growing number of modules and configurations efficiently. + +### Configuration Layers + +Configurations are merged and resolved in the following order of precedence: + +1. **Global Defaults**: Baseline settings applicable to all modules. +2. **Environment Variables**: Settings specific to the deployment environment. +3. **Stack-Specific Overrides**: Customizations for specific Pulumi stacks. +4. **User-Provided Configurations**: Inputs provided by users or module maintainers. +5. **Module-Specific Settings**: Configurations defined within individual modules. + +--- + +## Deployment Orchestration + +The `DeploymentManager` class in `deployment.py` handles the orchestration of module deployments: + +- **Module Discovery**: Automatically identifies modules to deploy based on the directory structure or configuration. +- **Initialization**: Loads module configurations and initializes resources. +- **Dependency Resolution**: Determines the order of resource creation based on dependencies. +- **Execution**: Coordinates the creation, updating, or deletion of resources as defined in the code. +- **Error Handling**: Implements mechanisms to catch and handle errors gracefully, including rollbacks if necessary. + +--- + +## Metadata and Tagging + +Consistent metadata and tagging are crucial for: + +- **Compliance**: Meeting organizational and regulatory requirements. +- **Traceability**: Tracking resource changes and deployments. +- **Management**: Facilitating resource identification and categorization. + +The `MetadataManager` provides methods to generate and apply standard tags to resources automatically. + +--- + +## Resource Management + +The `ResourceManager` in `resource_helpers.py` simplifies resource creation by: + +- **Applying Standard Configurations**: Ensuring resources are created with necessary settings. +- **Managing Metadata**: Automatically adding standard tags and labels. +- **Handling Dependencies**: Allowing explicit specification of resource dependencies. +- **Error Handling**: Providing consistent error messages and handling strategies. + +--- + +## Type Definitions and Schemas + +Using `TypedDict` and type annotations: + +- **Enhances Type Safety**: Catches errors during development rather than at runtime. +- **Improves Readability**: Makes code self-documenting and easier to understand. +- **Assists IDEs**: Provides better auto-completion and linting support. +- **Facilitates Validation**: Ensures data structures conform to expected formats. + +--- + +## Utility Functions + +Utility functions in `utils.py` support the core module by: + +- **Simplifying Common Tasks**: Providing reusable functions for tasks like configuration merging. +- **Enhancing Performance**: Implementing efficient algorithms where applicable. +- **Promoting Code Reuse**: Reducing duplication and promoting DRY principles. + +--- + +## Usage Guidelines + +### When to Use the Core Module + +Use the core module when you need to: + +- **Implement Shared Functionality**: Features or patterns used across multiple modules. +- **Enforce Standards**: Organizational policies, compliance requirements, or best practices. +- **Promote Consistency**: Ensure uniform behavior and configurations across resources. +- **Reduce Duplication**: Avoid repeating code by centralizing common logic. + +### Extending the Core Module + +When extending the core module: + +- **Assess Necessity**: Determine if the functionality is broadly applicable. +- **Maintain Separation of Concerns**: Keep functionalities modular and focused. +- **Ensure Backward Compatibility**: Avoid breaking existing implementations. +- **Document Thoroughly**: Update documentation and provide examples for new features. + +--- + +## Best Practices + +- **Type Annotations**: Use type hints throughout the code for clarity and error prevention. +- **Documentation**: Provide comprehensive docstrings and inline comments. +- **Compliance Adherence**: Utilize the core module's compliance features to meet requirements. +- **Modular Design**: Keep code organized and modular for maintainability. +- **Testing**: Write unit and integration tests to ensure code reliability. +- **DRY Principle**: Avoid code duplication by abstracting common functionalities. + +--- + +## Development Guidelines + +- **Code Style**: Follow the project's coding standards as outlined in the [Pulumi Python Development Guide](../developer_guide/pulumi-python.md). +- **Type Safety**: Enforce strict type checking using tools like Pyright. +- **Error Handling**: Implement robust error handling and provide meaningful messages. +- **Version Control**: Use meaningful commit messages and follow branching strategies. +- **Continuous Integration**: Integrate automated testing and linting in the development workflow. + +--- + +## Contribution Guidelines + +- **Proposing Changes**: + + - Discuss proposed changes with the maintainers before implementation. + - Ensure alignment with the project's goals and standards. + +- **Implementation**: + + - Write clean, type-safe code. + - Include unit tests and documentation updates. + - Follow the project's style and contribution guidelines. + +- **Pull Requests**: + + - Provide detailed descriptions of changes. + - Reference related issues or discussions. + - Be responsive to feedback during the review process. + +--- + +## Conclusion + +The Konductor Core Module is a vital component of the platform, enabling efficient, consistent, and compliant infrastructure deployments. By understanding its architecture and following the guidelines provided, developers can effectively utilize and contribute to the core module, fostering a collaborative and high-quality development environment. + +--- + +## Related Documentation + +- [Developer Guide](../developer_guide/README.md) +- [Pulumi Python Development Guide](../developer_guide/pulumi-python.md) +- [Documentation Style Guide](../developer_guide/documentation.md) +- [Contribution Guidelines](../contribution_guidelines.md) +- [TypedDict Reference Guide](../reference/TypedDict.md) diff --git a/docs/pulumi_iac.md b/docs/pulumi_iac.md deleted file mode 100644 index 3dccb91..0000000 --- a/docs/pulumi_iac.md +++ /dev/null @@ -1,168 +0,0 @@ -## Implementing Infrastructure as Code with Pulumi and GitOps - -### Introduction to Pulumi - -**Pulumi** is an open-source infrastructure as code (IaC) tool that enables developers to define and manage cloud resources using general-purpose programming languages like Python. Unlike traditional IaC tools that use domain-specific languages (DSLs), Pulumi leverages the full power of Python, allowing for advanced abstractions, code reuse, and integration with existing software development workflows. - -#### Why Choose Pulumi for IaC with Python? - -- **Unified Language**: Use Python for both application code and infrastructure definitions, reducing context switching. -- **Rich Ecosystem**: Access Python's extensive libraries and frameworks to enhance infrastructure code. -- **Advanced Abstractions**: Implement complex logic, loops, and conditional statements natively. -- **Team Collaboration**: Align development and operations teams by using a common language and tooling. - -#### Benefits Across Stakeholders - -- **Developers**: Simplify infrastructure management with familiar programming constructs. -- **DevOps Practitioners**: Streamline deployment processes and integrate with CI/CD pipelines. -- **Executive Leadership**: Accelerate time-to-market, enhance reliability, and optimize costs through automation. - -### Developing Pulumi Python Infrastructure as Code - -#### Getting Started with Pulumi and Python - -1. **Install Pulumi CLI**: Download and install the Pulumi command-line interface. -2. **Configure Cloud Provider Credentials**: Set up authentication for your target cloud platform (e.g., AWS, Azure, GCP). -3. **Initialize a New Project**: - - ```bash - pulumi new aws-python - ``` - -4. **Define Infrastructure in `__main__.py`**: Write Python code to declare cloud resources. - -#### Defining Infrastructure Resources - -Use Pulumi's Python SDK to create and configure resources. - -```python -import pulumi -from pulumi_aws import s3 - -# Create an S3 bucket -bucket = s3.Bucket('my-bucket', acl='private') - -# Export the bucket name -pulumi.export('bucket_name', bucket.id) -``` - -- **Resource Arguments**: Pass parameters to customize resources. -- **Outputs**: Export resource attributes for use in other components or stacks. - -#### State Management in Pulumi - -Pulumi maintains a **state file** that tracks the desired and actual state of resources. - -- **Backends**: Store state locally or in remote backends like Pulumi Service or AWS S3. -- **State Updates**: Pulumi compares the code with the state file to determine necessary changes. -- **Concurrent Access**: Locking mechanisms prevent simultaneous modifications. - -#### Modularization and Reusability in Pulumi - -Organize code into modules for better maintainability. - -- **Create Reusable Components**: Encapsulate resource definitions in classes or functions. - - ```python - class WebServer(pulumi.ComponentResource): - def __init__(self, name, opts=None): - super().__init__('custom:resource:WebServer', name, {}, opts) - # Define resources here - self.register_outputs({}) - ``` - -- **Parameterization**: Allow modules to accept inputs for flexibility. -- **Packaging**: Distribute modules as Python packages within your organization. - -### GitOps Workflow with Pulumi and Python - -#### Understanding GitOps - -**GitOps** is a workflow that uses Git repositories as the single source of truth for declarative infrastructure and applications. Changes to the infrastructure are made via code commits, triggering automated deployment processes. - -#### Implementing GitOps with Pulumi - -1. **Version Control**: Store all Pulumi code in a Git repository. -2. **Branching Strategy**: Use feature branches for development and pull requests for code reviews. -3. **Automated Pipelines**: Set up CI/CD pipelines to deploy changes upon merge. - - - **Continuous Integration (CI)**: Linting, testing, and validating infrastructure code. - - **Continuous Deployment (CD)**: Automatically apply infrastructure changes using Pulumi. - -4. **Pull Request Workflows**: - - - **Code Review**: Ensure code quality and adherence to standards. - - **Approval Gates**: Implement manual approvals for critical environments. - -#### CI/CD Pipeline Configuration - -- **Pipeline Steps**: - 1. **Checkout Code**: Retrieve the latest code from the repository. - 2. **Install Dependencies**: Set up the Python environment and install Pulumi packages. - 3. **Login to Pulumi Backend**: Authenticate with the state backend. - 4. **Preview Changes**: Run `pulumi preview` to show potential changes. - 5. **Apply Changes**: Execute `pulumi up` to update infrastructure. - -- **Sample Pipeline Configuration** (e.g., using GitHub Actions): - - ```yaml - name: Pulumi CI/CD - - on: - push: - branches: - - main - - jobs: - deploy: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: '3.x' - - name: Install Dependencies - run: pip install -r requirements.txt - - name: Install Pulumi - run: curl -fsSL https://get.pulumi.com | sh - - name: Login to Pulumi Backend - run: pulumi login - - name: Preview Changes - run: pulumi preview - - name: Apply Changes - run: pulumi up --yes - ``` - -#### Best Practices for GitOps with Pulumi - -- **Immutable Infrastructure**: Avoid manual changes to deployed resources. -- **Declarative Definitions**: Ensure all infrastructure configurations are code-driven. -- **Audit Trails**: Utilize Git history for change tracking and compliance. -- **Rollback Mechanisms**: Revert to previous states by checking out earlier commits. - -### Advantages for Executive Leadership - -#### Accelerated Delivery and Agility - -- **Faster Deployments**: Automation reduces time between development and production. -- **Market Responsiveness**: Quickly adapt infrastructure to changing business needs. - -#### Risk Mitigation and Compliance - -- **Consistency**: Code-driven deployments minimize human error. -- **Traceability**: Detailed logs and version control aid in audits and compliance checks. - -#### Cost Optimization - -- **Resource Efficiency**: Automate scaling and teardown of resources to match demand. -- **Operational Overhead Reduction**: Streamline processes reduce manual labor costs. - -#### Strategic Alignment - -- **DevOps Culture**: Foster collaboration between development and operations teams. -- **Innovation Enablement**: Free up teams to focus on value-adding activities rather than manual tasks. - -### Conclusion - -Integrating Pulumi with GitOps workflows empowers organizations to manage infrastructure with the same rigor and agility as application code. By utilizing Python for IaC, teams benefit from a cohesive development experience, advanced abstractions, and seamless integration with existing tools and processes. This approach not only enhances technical efficiency but also delivers strategic advantages that align with organizational goals. diff --git a/docs/reference/TypeDict.md b/docs/reference/TypeDict.md deleted file mode 100644 index f859c8a..0000000 --- a/docs/reference/TypeDict.md +++ /dev/null @@ -1,368 +0,0 @@ -# TypedDict Reference Guide for Konductor - -## Introduction - -This reference guide provides comprehensive documentation on using `TypedDict` within the Konductor Infrastructure as Code (IaC) platform. `TypedDict`, introduced in PEP 589, is central to Konductor's configuration management strategy, providing type-safe dictionary structures that enhance code reliability and maintainability. - -## Table of Contents - -1. [Overview](#overview) - - [Why TypedDict?](#why-typeddict) - - [Benefits in Konductor](#benefits-in-konductor) - - [Integration with Pulumi](#integration-with-pulumi) - -2. [Using TypedDict in Konductor](#using-typeddict-in-konductor) - - [Basic Usage](#basic-usage) - - [Advanced Patterns](#advanced-patterns) - - [Best Practices](#best-practices) - -3. [Configuration Management](#configuration-management) - - [Module Configurations](#module-configurations) - - [Default Values](#default-values) - - [Configuration Validation](#configuration-validation) - -4. [Type Checking](#type-checking) - - [Static Analysis with Pyright](#static-analysis-with-pyright) - - [Runtime Considerations](#runtime-considerations) - - [Common Type Issues](#common-type-issues) - -5. [Examples](#examples) - - [Basic Examples](#basic-examples) - - [Module Examples](#module-examples) - - [Complex Configurations](#complex-configurations) - -6. [Troubleshooting](#troubleshooting) - - [Common Issues](#common-issues) - - [Best Practices](#best-practices-1) - - [FAQs](#faqs) - -## Overview - -### Why TypedDict? - -`TypedDict` provides several key advantages for configuration management in Konductor: - -- **Type Safety**: Catch configuration errors at development time -- **IDE Support**: Enhanced autocompletion and type hints -- **Documentation**: Self-documenting configuration structures -- **Validation**: Static type checking for configurations -- **Maintainability**: Clear contract for configuration objects - -### Benefits in Konductor - -In Konductor, `TypedDict` is used to: - -1. Define module configurations -2. Specify resource properties -3. Structure complex data hierarchies -4. Ensure configuration consistency -5. Facilitate static analysis - -### Integration with Pulumi - -`TypedDict` integrates seamlessly with Pulumi's Python SDK: - -```python -from typing import TypedDict, List -from pulumi import ResourceOptions - -class ResourceConfig(TypedDict): - name: str - tags: dict - opts: ResourceOptions -``` - -## Using TypedDict in Konductor - -### Basic Usage - -Define configuration schemas using `TypedDict`: - -```python -from typing import TypedDict, Optional - -class ModuleConfig(TypedDict): - enabled: bool - version: Optional[str] - namespace: str - labels: dict - -# Default configuration -module_defaults: ModuleConfig = { - "enabled": False, - "version": None, - "namespace": "default", - "labels": {} -} -``` - -### Advanced Patterns - -#### Nested Configurations - -```python -class ContainerPort(TypedDict): - containerPort: int - protocol: str - -class Container(TypedDict): - name: str - image: str - ports: List[ContainerPort] - -class PodSpec(TypedDict): - containers: List[Container] -``` - -#### Optional Fields - -```python -from typing import TypedDict, Optional - -class AwsConfig(TypedDict, total=False): - region: str # Required - profile: Optional[str] # Optional - tags: dict # Optional -``` - -### Best Practices - -1. **Use Clear Names**: - ```python - # Good - class NetworkConfig(TypedDict): - vpc_cidr: str - subnet_cidrs: List[str] - - # Avoid - class Config(TypedDict): - cidr: str - subnets: List[str] - ``` - -2. **Document Fields**: - ```python - class DeploymentConfig(TypedDict): - """Configuration for Kubernetes deployments. - - Attributes: - name: Name of the deployment - replicas: Number of desired replicas - image: Container image to deploy - """ - name: str - replicas: int - image: str - ``` - -3. **Provide Defaults**: - ```python - deployment_defaults: DeploymentConfig = { - "name": "app", - "replicas": 1, - "image": "nginx:latest" - } - ``` - -## Configuration Management - -### Module Configurations - -Each module defines its configuration schema: - -```python -# modules/aws/types.py -class AwsModuleConfig(TypedDict): - enabled: bool - region: str - profile: Optional[str] - tags: Dict[str, str] - -# Default configuration -aws_defaults: AwsModuleConfig = { - "enabled": False, - "region": "us-west-2", - "profile": None, - "tags": {} -} -``` - -### Default Values - -Implement default value handling: - -```python -def merge_with_defaults( - user_config: dict, - defaults: TypedDict -) -> TypedDict: - """Merge user configuration with defaults.""" - config = defaults.copy() - config.update(user_config) - return config -``` - -### Configuration Validation - -Use Pyright for static validation: - -```json -# Configure in pyrightconfig.json -{ - "typeCheckingMode": "strict", - "reportUnknownMemberType": true -} -``` - -## Type Checking - -### Static Analysis with Pyright - -Enable strict type checking: - -```json -{ - "include": ["/.py"], - "exclude": ["/_pycache_/"], - "reportMissingImports": true, - "pythonVersion": "3.8", - "typeCheckingMode": "strict" -} -``` - -### Runtime Considerations - -`TypedDict` performs no runtime validation: - -```python -from typing import cast - -def load_config(config_dict: dict) -> ModuleConfig: - """Load and validate configuration. - Note: TypedDict casting provides no runtime checks. - Add explicit validation if needed. - """ - config = cast(ModuleConfig, config_dict) - validate_config(config) # Add custom validation if needed - return config -``` - -### Common Type Issues - -1. **Missing Required Fields**: - ```python - # Error: Missing required 'name' field - config: ModuleConfig = { - "enabled": True - } - ``` - -2. **Incorrect Types**: - ```python - # Error: 'enabled' must be bool - config: ModuleConfig = { - "enabled": "true", # Should be True - "name": "test" - } - ``` - -## Examples - -### Basic Examples - -```python -from typing import TypedDict, List - -class ServiceConfig(TypedDict): - name: str - port: int - replicas: int - -# Usage -service_config: ServiceConfig = { - "name": "web", - "port": 80, - "replicas": 3 -} -``` - -### Module Examples - -```python -# AWS VPC Configuration -class VpcConfig(TypedDict): - cidr_block: str - enable_dns: bool - tags: Dict[str, str] - -# Kubernetes Deployment -class DeploymentConfig(TypedDict): - name: str - namespace: str - replicas: int - image: str - ports: List[int] -``` - -### Complex Configurations - -```python -class DatabaseConfig(TypedDict): - engine: str - version: str - size: str - backup: bool - -class ApplicationConfig(TypedDict): - name: str - environment: str - replicas: int - database: DatabaseConfig - features: Dict[str, bool] -``` - -## Troubleshooting - -### Common Issues - -1. **Type Mismatch Errors**: - ```python - # Error: Type mismatch - config: NetworkConfig = { - "vpc_cidr": 10, # Should be str - "subnet_cidrs": ["10.0.0.0/24"] - } - ``` - -2. **Missing Fields**: - ```python - # Error: Missing required field - config: ServiceConfig = { - "name": "api" # Missing required 'port' field - } - ``` - -### Best Practices - -1. Use explicit type annotations -2. Provide default values -3. Document configuration schemas -4. Use static type checking -5. Implement custom validation when needed - -### FAQs - -**Q: When should I use TypedDict vs. dataclass?** -A: Use TypedDict when working with dictionary-like structures, especially for configurations. Use dataclasses for more complex objects with methods. - -**Q: How do I handle optional fields?** -A: Use `total=False` or wrap types with `Optional[]`. - -**Q: Can I use TypedDict with Pulumi outputs?** -A: Yes, but be aware that Pulumi outputs are handled differently than regular values. - -## Related Documentation - -- [Python Development Standards](./PULUMI_PYTHON.md) -- [Style Guide](./style_guide.md) -- [Developer Guide](../developer_guide/konductor_developer_guide.md) diff --git a/docs/reference/best_practices.md b/docs/reference/best_practices.md deleted file mode 100644 index 13566e3..0000000 --- a/docs/reference/best_practices.md +++ /dev/null @@ -1,211 +0,0 @@ -# Best Practices for Clean, Readable IaC Code in Pulumi Python - -This document offers Pulumi Python Infrastructure as Code (IaC) project coding practices that enhance code readability, maintainability, and efficiency. - -## Table of Contents - -1. [Introduction](#introduction) -2. [Using \*args and \*\*kwargs in Resource Definitions](#using-args-and-kwargs-in-resource-definitions) -3. [Leveraging List Comprehensions for Resource Collections](#leveraging-list-comprehensions-for-resource-collections) -4. [Utilizing `zip()` for Parallel Resource Configuration](#utilizing-zip-for-parallel-resource-configuration) -5. [Merging Dictionaries for Resource Arguments](#merging-dictionaries-for-resource-arguments) -6. [Chaining Comparison Operators in Conditional Statements](#chaining-comparison-operators-in-conditional-statements) -7. [Simplifying Conditional Assignments with Ternary Operators](#simplifying-conditional-assignments-with-ternary-operators) -8. [Enhancing Functionality with Decorators](#enhancing-functionality-with-decorators) -9. [Conclusion](#conclusion) - -## Introduction - -Infrastructure as Code (IaC) allows us to manage and provision infrastructure through code rather than manual processes. As we develop our IaC project with Pulumi Python, adopting clean and readable code practices is crucial. This ensures that our codebase is maintainable, efficient, and accessible to other developers. - -## Using \*args and \*\*kwargs in Resource Definitions - -In Python, `*args` and `**kwargs` are used to pass a variable number of arguments to functions. In the context of Pulumi, they can make resource definitions more flexible and reusable. - -### Example: Defining Flexible Resource Creation Functions - -```python -from typing import Any, Dict -import pulumi -import pulumi_aws as aws - -def create_security_group(name: str, *ingress_rules: Dict[str, Any], **kwargs): - return aws.ec2.SecurityGroup( - resource_name=name, - ingress=list(ingress_rules), - **kwargs - ) - -# Usage -sg = create_security_group( - "web-sg", - {"protocol": "tcp", "from_port": 80, "to_port": 80, "cidr_blocks": ["0.0.0.0/0"]}, - {"protocol": "tcp", "from_port": 443, "to_port": 443, "cidr_blocks": ["0.0.0.0/0"]}, - tags={"Environment": "Production"} -) -``` - -**Benefits:** - -- Allows passing any number of ingress rules without modifying the function signature. -- Enables flexible resource creation with additional arguments via `**kwargs`. - -## Leveraging List Comprehensions for Resource Collections - -List comprehensions provide a concise way to create lists. They are particularly useful when creating multiple similar resources in Pulumi. - -### Example: Creating Multiple S3 Buckets - -```python -import pulumi_aws as aws - -bucket_names = ["logs", "data", "backup"] - -buckets = [ - aws.s3.Bucket( - f"{name}-bucket", - bucket=f"mycompany-{name}-bucket" - ) for name in bucket_names -] -``` - -**Benefits:** - -- Reduces code verbosity by eliminating explicit loops. -- Enhances readability by keeping related code in a single expression. - -## Utilizing `zip()` for Parallel Resource Configuration - -The `zip()` function allows you to iterate over multiple iterables in parallel. In Pulumi, this is useful when you need to configure resources that depend on multiple lists of values. - -Note: This sample code is an example only and may not be a valid pattern for authoring instances. Use this pattern only when appropriate and optimal. - -### Example: Creating EC2 Instances with Specific Configurations - -```python -import pulumi_aws as aws - -instance_types = ["t2.micro", "t2.small", "t2.medium"] -ami_ids = ["ami-123", "ami-456", "ami-789"] -names = ["web-server", "app-server", "db-server"] - -instances = [ - aws.ec2.Instance( - f"{name}-instance", - instance_type=instance_type, - ami=ami_id, - tags={"Name": name} - ) - for instance_type, ami_id, name in zip(instance_types, ami_ids, names) -] -``` - -**Benefits:** - -- Simplifies the creation of resources with paired configurations. -- Ensures that corresponding elements from multiple lists are used together. - -## Merging Dictionaries for Resource Arguments - -Merging dictionaries is common when combining default configurations with environment-specific settings. In Pulumi, this helps in managing resource arguments. - -### Example: Merging Default and Custom Tags - -```python -default_tags = {"Project": "MyProject", "ManagedBy": "Pulumi"} -environment_tags = {"Environment": "Production"} - -all_tags = {**default_tags, **environment_tags} - -bucket = aws.s3.Bucket( - "my-bucket", - tags=all_tags -) -``` - -**Benefits:** - -- Avoids repetitive code by reusing common configurations. -- Enhances maintainability by centralizing default settings. - -## Chaining Comparison Operators in Conditional Statements - -Chaining comparison operators allows for more concise and readable conditional statements, which is useful in resource validation and conditional resource creation. - -### Example: Validating Resource Parameters - -```python -def validate_instance_count(count): - if 1 <= count <= 5: - return True - else: - raise ValueError("Instance count must be between 1 and 5.") - -# Usage -validate_instance_count(3) # Valid -validate_instance_count(6) # Raises ValueError -``` - -**Benefits:** - -- Improves readability by reducing the need for multiple logical operators. -- Makes validation logic more intuitive. - -## Simplifying Conditional Assignments with Ternary Operators - -Ternary operators allow for inline conditional assignments, making code more concise. - -### Example: Setting Resource Names Based on Environment - -```python -environment = pulumi.get_stack() # e.g., 'dev' or 'prod' - -db_name = "mydb-prod" if environment == "prod" else "mydb-dev" - -rds_instance = aws.rds.Instance( - "db-instance", - instance_class="db.t2.micro", - allocated_storage=20, - name=db_name -) -``` - -**Benefits:** - -- Reduces code lines for simple conditional assignments. -- Enhances readability by keeping related logic together. - -## Enhancing Functionality with Decorators - -Decorators in Python allow you to modify the behavior of functions or methods. In Pulumi, decorators can be used to add common functionality such as logging, error handling, or input validation. - -### Example: Creating a Decorator for Resource Creation Logging - -```python -import functools -import logging - -def log_resource_creation(func): - @functools.wraps(func) - def wrapper(*args, **kwargs): - resource_name = kwargs.get('resource_name', 'Unknown') - logging.info(f"Creating resource: {resource_name}") - result = func(*args, **kwargs) - logging.info(f"Resource created: {resource_name}") - return result - return wrapper - -# Applying the decorator -@log_resource_creation -def create_bucket(resource_name, **kwargs): - return aws.s3.Bucket(resource_name, **kwargs) - -# Usage -bucket = create_bucket("my-logging-bucket") -``` - -**Benefits:** - -- Adds cross-cutting concerns without modifying the original function logic. -- Promotes code reusability and cleaner function definitions. - diff --git a/docs/reference/pulumi_python.md b/docs/reference/pulumi_python.md deleted file mode 100644 index b94ad41..0000000 --- a/docs/reference/pulumi_python.md +++ /dev/null @@ -1,475 +0,0 @@ -# Pulumi Python Development Standards - -## Table of Contents - -1. [Introduction](#introduction) -2. [Development Philosophy](#development-philosophy) -3. [Environment Setup](#environment-setup) -4. [Code Organization](#code-organization) -5. [Type Safety](#type-safety) -6. [Configuration Management](#configuration-management) -7. [Development Standards](#development-standards) -8. [Testing Requirements](#testing-requirements) -9. [Documentation Requirements](#documentation-requirements) -10. [Best Practices](#best-practices) -11. [Appendices](#appendices) - -## Introduction - -This document establishes the development standards and best practices for Pulumi Python projects within the Konductor framework. It serves as the authoritative reference for code quality, maintainability, and consistency across all modules and components. - -### Core Principles - -- **Type Safety**: Enforce static type checking for reliability -- **Maintainability**: Write clear, documented, and testable code -- **Consistency**: Follow established patterns and standards -- **Quality**: Prioritize code quality over feature quantity - -## Development Philosophy - -> "Features are nice. Quality is paramount." - -Our development approach emphasizes: - -1. **Code Quality**: - - Comprehensive type checking - - Clear documentation - - Thorough testing - - Consistent style - -2. **Developer Experience**: - - Intuitive interfaces - - Clear error messages - - Helpful documentation - - Streamlined workflows - -3. **Maintainability**: - - Modular design - - Single responsibility - - Clear dependencies - - Consistent patterns - -## Environment Setup - -### Prerequisites - -- Python 3.8+ -- Poetry for dependency management -- Pulumi CLI -- Pyright for type checking -- Git for version control - -### Initial Setup - -1. **Install Poetry**: - ```bash - curl -sSL https://install.python-poetry.org | python3 - - ``` - -2. **Configure Poetry**: - ```bash - poetry config virtualenvs.in-project true - ``` - -3. **Initialize Project**: - ```bash - poetry install - poetry shell - ``` - -4. **Configure Pulumi**: - ```yaml - # Pulumi.yaml - name: your-project - runtime: - name: python - options: - toolchain: poetry - typechecker: pyright - ``` - -## Code Organization - -### Directory Structure - -``` -project/ -├── pulumi/ -│ ├── __main__.py -│ ├── core/ -│ │ ├── __init__.py -│ │ ├── config.py -│ │ └── utils.py -│ └── modules/ -│ ├── aws/ -│ │ ├── __init__.py -│ │ ├── types.py -│ │ └── deploy.py -│ └── kubernetes/ -│ ├── __init__.py -│ ├── types.py -│ └── deploy.py -├── tests/ -├── pyproject.toml -├── poetry.lock -└── pyrightconfig.json -``` - -### Module Structure - -Each module should follow this structure: - -```python -# types.py -from typing import TypedDict - -class ModuleConfig(TypedDict): - """Configuration schema for the module.""" - enabled: bool - version: str - parameters: Dict[str, Any] - -# deploy.py -def deploy_module( - config: ModuleConfig, - dependencies: List[Resource] -) -> Resource: - """Deploy module resources.""" - pass -``` - -## Type Safety - -### TypedDict Usage - -1. **Configuration Schemas**: - ```python - from typing import TypedDict, Optional - - class NetworkConfig(TypedDict): - """Network configuration schema.""" - vpc_cidr: str - subnet_cidrs: List[str] - enable_nat: bool - tags: Optional[Dict[str, str]] - ``` - -2. **Default Values**: - ```python - network_defaults: NetworkConfig = { - "vpc_cidr": "10.0.0.0/16", - "subnet_cidrs": [], - "enable_nat": True, - "tags": None - } - ``` - -### Type Checking - -1. **Configure Pyright**: - ```json - { - "include": ["**/*.py"], - "exclude": ["**/__pycache__/**"], - "reportMissingImports": true, - "pythonVersion": "3.8", - "typeCheckingMode": "strict" - } - ``` - -2. **Run Type Checking**: - ```bash - poetry run pyright - ``` - -## Configuration Management - -### Configuration Structure - -1. **Module Configurations**: - ```python - class ModuleConfig(TypedDict): - enabled: bool - version: str - parameters: Dict[str, Any] - - def load_config( - module_name: str, - config: pulumi.Config - ) -> ModuleConfig: - """Load and validate module configuration.""" - pass - ``` - -2. **Validation**: - ```python - def validate_config(config: ModuleConfig) -> None: - """Validate configuration values.""" - if not isinstance(config["enabled"], bool): - raise TypeError("enabled must be a boolean") - ``` - -## Development Standards - -### Code Style - -1. **Naming Conventions**: - - Classes: `PascalCase` - - Functions/Variables: `snake_case` - - Constants: `UPPER_SNAKE_CASE` - - Private members: `_leading_underscore` - -2. **Documentation**: - ```python - def create_resource( - name: str, - config: ResourceConfig - ) -> Resource: - """Create a new resource. - - Args: - name: Resource name - config: Resource configuration - - Returns: - Created resource - - Raises: - ValueError: If configuration is invalid - """ - pass - ``` - -3. **Error Handling**: - ```python - try: - resource = create_resource(name, config) - except ValueError as e: - pulumi.log.error(f"Failed to create resource: {e}") - raise - ``` - -### Function Signatures - -1. **Type Annotations**: - ```python - from typing import Optional, List, Dict, Any - - def deploy_resources( - configs: List[ResourceConfig], - dependencies: Optional[List[Resource]] = None, - **kwargs: Any - ) -> List[Resource]: - """Deploy multiple resources.""" - pass - ``` - -2. **Return Types**: - ```python - def get_resource_status( - resource_id: str - ) -> Optional[Dict[str, Any]]: - """Get resource status or None if not found.""" - pass - ``` - -## Testing Requirements - -### Unit Tests - -1. **Test Structure**: - ```python - import pytest - from typing import Generator - - @pytest.fixture - def resource_config() -> Generator[ResourceConfig, None, None]: - """Provide test configuration.""" - config = create_test_config() - yield config - cleanup_test_config(config) - - def test_resource_creation( - resource_config: ResourceConfig - ) -> None: - """Test resource creation.""" - result = create_resource("test", resource_config) - assert result.id is not None - ``` - -2. **Mocking**: - ```python - @pytest.fixture - def mock_aws_client(mocker): - """Mock AWS client.""" - return mocker.patch("boto3.client") - ``` - -### Integration Tests - -```python -def test_module_deployment( - pulumi_stack: auto.Stack -) -> None: - """Test full module deployment.""" - result = pulumi_stack.up() - assert result.summary.resource_changes["create"] > 0 -``` - -## Documentation Requirements - -### Code Documentation - -1. **Module Documentation**: - ```python - """AWS networking module. - - This module manages AWS networking resources including VPCs, - subnets, and security groups. - - Example: - ```python - config = NetworkConfig(...) - vpc = create_vpc(config) - ``` - """ - ``` - -2. **Function Documentation**: - ```python - def create_vpc( - config: NetworkConfig, - tags: Optional[Dict[str, str]] = None - ) -> Resource: - """Create a VPC with the specified configuration. - - Args: - config: VPC configuration - tags: Optional resource tags - - Returns: - Created VPC resource - - Raises: - ValueError: If CIDR is invalid - """ - pass - ``` - -## Best Practices - -1. **Resource Management**: - ```python - def create_resources( - configs: List[ResourceConfig] - ) -> List[Resource]: - """Create multiple resources with proper cleanup.""" - resources = [] - try: - for config in configs: - resource = create_resource(config) - resources.append(resource) - return resources - except Exception: - cleanup_resources(resources) - raise - ``` - -2. **Configuration Handling**: - ```python - def load_config( - path: str, - defaults: Dict[str, Any] - ) -> Dict[str, Any]: - """Load configuration with defaults.""" - config = load_yaml(path) - return deep_merge(defaults, config) - ``` - -3. **Error Handling**: - ```python - class ResourceError(Exception): - """Base exception for resource operations.""" - pass - - class ResourceNotFoundError(ResourceError): - """Raised when a resource cannot be found.""" - pass - ``` - -## Appendices - -### A. Common Patterns - -1. **Resource Tags**: - ```python - def get_resource_tags( - name: str, - environment: str, - additional_tags: Optional[Dict[str, str]] = None - ) -> Dict[str, str]: - """Generate standard resource tags.""" - tags = { - "Name": name, - "Environment": environment, - "ManagedBy": "pulumi" - } - if additional_tags: - tags.update(additional_tags) - return tags - ``` - -2. **Resource Names**: - ```python - def generate_resource_name( - base_name: str, - suffix: Optional[str] = None - ) -> str: - """Generate consistent resource names.""" - name = f"{base_name}-{pulumi.get_stack()}" - if suffix: - name = f"{name}-{suffix}" - return name.lower() - ``` - -### B. Type Checking Examples - -```python -from typing import TypedDict, Optional, List, Dict, Any - -class ResourceConfig(TypedDict): - """Resource configuration.""" - name: str - type: str - parameters: Dict[str, Any] - tags: Optional[Dict[str, str]] - -def create_resource( - config: ResourceConfig, - dependencies: Optional[List[Resource]] = None -) -> Resource: - """Create a resource with type checking.""" - validate_config(config) - return Resource( - config["name"], - config["parameters"], - opts=ResourceOptions(depends_on=dependencies) - ) -``` - -### C. Testing Patterns - -```python -import pytest -from pulumi import automation as auto - -def test_infrastructure_deployment(): - """Test full infrastructure deployment.""" - stack = auto.create_stack(...) - try: - result = stack.up() - assert result.summary.result == "succeeded" - finally: - stack.destroy() - stack.workspace.remove_stack(stack.name) -``` diff --git a/docs/reference/style_guide.md b/docs/reference/style_guide.md deleted file mode 100644 index 4694bf6..0000000 --- a/docs/reference/style_guide.md +++ /dev/null @@ -1,267 +0,0 @@ -# Konductor Documentation Style Guide - -## Introduction - -This style guide establishes standards for creating and maintaining documentation within the Konductor project. It ensures consistency, clarity, and accessibility across all documentation while aligning with the project's technical standards outlined in `PULUMI_PYTHON.md`. - -## Table of Contents - -1. [General Principles](#general-principles) -2. [Document Structure](#document-structure) -3. [Writing Style](#writing-style) -4. [Formatting Standards](#formatting-standards) -5. [Code Examples](#code-examples) -6. [Links and References](#links-and-references) -7. [Images and Diagrams](#images-and-diagrams) -8. [Accessibility Guidelines](#accessibility-guidelines) -9. [Version Control](#version-control) -10. [File Organization](#file-organization) - -## General Principles - -### Clarity -- Write for your audience's knowledge level -- Define technical terms on first use -- Use consistent terminology throughout -- Avoid jargon unless necessary - -### Completeness -- Include all necessary information -- Provide context for technical concepts -- Link to related documentation -- Include troubleshooting guidance - -### Maintainability -- Keep content modular -- Use relative links -- Follow the DRY (Don't Repeat Yourself) principle -- Regular reviews and updates - -## Document Structure - -### Required Sections - -1. **Title**: Clear, descriptive title using H1 (`#`) -2. **Introduction**: Brief overview of the document's purpose -3. **Table of Contents**: For documents longer than 3 sections -4. **Prerequisites** (if applicable): Required knowledge or setup -5. **Main Content**: Organized in logical sections -6. **Conclusion** (if applicable): Summary or next steps -7. **Related Resources**: Links to related documentation - -### Header Hierarchy - -```markdown -# Document Title (H1) -## Section Title (H2) -### Sub-section Title (H3) -#### Minor Sub-section Title (H4) -``` - -### Metadata Block (Optional) - -```yaml ---- -title: Document Title -description: Brief description of the document -authors: [Author Name] -date: YYYY-MM-DD -version: 0.0.1 ---- -``` - -## Writing Style - -### Voice and Tone -- Use active voice -- Be direct and concise -- Maintain a professional, friendly tone -- Write in present tense - -### Paragraphs -- Keep paragraphs focused on a single topic -- Use short paragraphs (3-5 sentences) -- Include transition sentences between sections - -### Lists -- Use bullet points for unordered lists -- Use numbered lists for sequential steps -- Maintain parallel structure in list items - -## Formatting Standards - -### Text Formatting - -- **Bold**: Use for emphasis and UI elements -- *Italic*: Use for introducing new terms -- `Code`: Use for code snippets, file names, and commands -- ***Bold Italic***: Avoid unless absolutely necessary - -### Code Blocks - -- Use triple backticks with language identifier -- Include description of code's purpose -- Add line numbers for longer snippets - -```python -# Example code block -def example_function(): - """Docstring describing the purpose of the function.""" - pass -``` - -### Tables - -- Use tables for structured data -- Include header row -- Align columns consistently - -| Header 1 | Header 2 | Header 3 | -|----------|----------|----------| -| Data | Data | Data | - -## Code Examples - -### General Guidelines - -- Keep examples simple and focused -- Include comments explaining key concepts -- Use meaningful variable and function names -- Follow `PULUMI_PYTHON.md` coding standards - -### Example Structure - -```python -from typing import TypedDict, Optional - -class ConfigExample(TypedDict): - """Example configuration structure. - Attributes: - name: Resource name - enabled: Whether the resource is enabled - """ - name: str - enabled: bool - -def example_function(config: ConfigExample) -> None: - """Example function with type annotations. - Args: - config: Configuration dictionary - """ - if config.get("enabled", True): - print(f"Resource {config['name']} is enabled.") -``` - -## Links and References - -### Internal Links -- Use relative paths -- Link to specific sections where applicable -- Check links regularly for validity - -### External Links -- Include link text that makes sense out of context -- Add notes for external dependencies -- Consider link stability - -### Cross-References -- Use consistent terminology -- Link to glossary terms -- Reference related documentation - -## Images and Diagrams - -### Requirements -- Use SVG format when possible -- Include alt text for accessibility -- Provide high-resolution versions -- Keep file sizes reasonable - -### Captions -- Include descriptive captions -- Number figures sequentially -- Reference figures in text - -## Accessibility Guidelines - -### Text Content -- Use sufficient color contrast -- Avoid relying solely on color -- Provide text alternatives for media -- Use semantic markup - -### Navigation -- Logical heading structure -- Descriptive link text -- Keyboard navigation support -- Skip navigation links - -### Media -- Alt text for images -- Transcripts for audio -- Captions for video -- Accessible data tables - -## Version Control - -### Commit Messages -- Use clear, descriptive messages -- Reference issue numbers -- Indicate documentation changes -- Follow conventional commits - -Example: - -``` -docs(aws): update EKS cluster setup guide -``` - -* Add troubleshooting section -* Update configuration examples -* Fix broken links - -Fixes: -- #123: Fixed broken link to EKS documentation - - -### Branching -- Create feature branches for substantial changes -- Use `docs/` prefix for documentation branches -- Review changes before merging -- Keep documentation in sync with code - -## File Organization - -### Naming Conventions -- Use lowercase with underscores -- Be descriptive but concise -- Include relevant prefixes -- Maintain consistent extensions - -### Directory Structure -- Follow the established hierarchy -- Group related documents -- Use README files for navigation -- Maintain clean organization - -### File Templates -- Use standard templates -- Include required sections -- Maintain consistent structure -- Update templates as needed - ---- - -## Implementation Notes - -This style guide should be: -- Referenced during documentation creation -- Updated based on team feedback -- Reviewed quarterly -- Enforced through automation where possible - -## Related Documents - -- [PULUMI_PYTHON.md](./PULUMI_PYTHON.md) -- [TypedDict.md](./TypedDict.md) -- [Contribution Guidelines](../developer_guide/contribution_guidelines.md) diff --git a/docs/roadmaps/ROADMAP.md b/docs/roadmaps/ROADMAP.md deleted file mode 100644 index 20539d5..0000000 --- a/docs/roadmaps/ROADMAP.md +++ /dev/null @@ -1,218 +0,0 @@ -# Next-Generation Platform Engineering Roadmap - -## Table of Contents - -1. [Introduction](#introduction) -2. [Vision and Goals](#vision-and-goals) -3. [Strategic Pillars](#strategic-pillars) -4. [Technical Architecture](#technical-architecture) -5. [Implementation Phases](#implementation-phases) -6. [Development Standards](#development-standards) -7. [Key Components](#key-components) -8. [Success Metrics](#success-metrics) -9. [Timeline and Milestones](#timeline-and-milestones) -10. [Appendices](#appendices) - -## Introduction - -The Konductor Platform Engineering initiative aims to establish a cloud-agnostic, compliance-ready infrastructure platform that accelerates development while maintaining security and governance. This roadmap outlines our journey from initial setup through full multi-cloud deployment. - -### Overview -- Architecture focuses on modular design with TypedDict-based configurations -- Emphasis on static type checking and automated compliance controls -- Integration points for future cloud provider expansion - -### Objectives -- Clear implementation patterns for new modules -- Standardized approach to configuration management -- Automated testing and validation frameworks - -## Standards -- Well-documented setup procedures -- Example-driven development guides -- Clear path for contribution - -## Vision and Goals - -### Primary Objectives -- Create a cloud-agnostic infrastructure platform -- Automate compliance and security controls -- Enable self-service for application teams -- Reduce time-to-production for new services - -### Key Outcomes -- Reduced manual compliance tasks -- Faster application deployment -- 99.99% automated infrastructure orchestration -- Zero-touch compliance validation - -## Strategic Pillars - -### 1. Cloud-Agnostic Architecture - -```python -# Example Configuration Structure -class CloudProviderConfig(TypedDict): - provider: str # aws, azure, or gcp - region: str - credentials: Dict[str, str] - compliance_level: str -``` - -### 2. Compliance Automation - -```python -# Example Compliance Integration -class ComplianceConfig(TypedDict): - nist_controls: List[str] - fisma_level: str - audit_logging: bool - encryption: Dict[str, str] -``` - -### 3. Developer Experience - -```python -# Example Module Structure -modules/ -├── aws/ -│ ├── types.py # TypedDict definitions -│ ├── deploy.py # Deployment logic -│ └── README.md # Module documentation -``` - -## Technical Architecture - -### Core Components - -1. **Configuration Management** - - TypedDict-based schemas - - Static type checking with Pyright - - Centralized configuration validation - -2. **Module System** - - Pluggable architecture - - Standard interfaces - - Automated testing framework - -3. **Compliance Framework** - - Policy as Code implementation - - Automated controls - - Audit logging and reporting - -## Implementation Phases - -### Phase 1: Foundations (Months 1-3) -- Set up IaC framework with Pulumi -- Implement core TypedDict configurations -- Establish CI/CD pipelines - -### Phase 2: Core Infrastructure (Months 4-6) -- Deploy base AWS infrastructure -- Implement networking modules -- Set up monitoring and logging - -### Phase 3: Compliance Integration (Months 7-9) -- Implement NIST controls -- Set up FISMA compliance -- Automate compliance reporting - -### Phase 4: Multi-Cloud Expansion (Months 10-12) -- Add Azure support -- Implement GCP integration -- Cross-cloud networking - -## Development Standards - -### Code Organization -```python -# Standard Module Structure -class ModuleConfig(TypedDict): - enabled: bool - version: str - parameters: Dict[str, Any] - -def deploy_module( - config: ModuleConfig, - dependencies: List[Resource] -) -> Resource: - """Deploy module with standard interface.""" - pass -``` - -### Testing Requirements -- Unit tests for all modules -- Integration tests for workflows -- Compliance validation tests - -## Key Components - -### 1. Account Structure -- Multi-account strategy -- Role-based access control -- Resource organization - -### 2. Infrastructure Components -- Networking -- Compute resources -- Storage solutions -- Security controls - -### 3. Compliance Framework -- Policy definitions -- Control mapping -- Audit mechanisms - -## Success Metrics - -### Technical Metrics -- Deployment success rate -- Infrastructure drift percentage -- Test coverage -- Type checking compliance - -### Business Metrics -- Time to deployment -- Cost optimization -- Compliance achievement -- Developer satisfaction - -## Timeline and Milestones - -- Complete Phase 1: Foundations -- Establish development standards -- Initial AWS implementation - -- Complete Phase 2: Core Infrastructure -- Deploy first production workload -- Achieve initial compliance targets - -- Complete Phase 3: Compliance Integration -- Full NIST compliance -- Automated security controls - -- Complete Phase 4: Multi-Cloud Expansion -- Azure and GCP integration -- Cross-cloud operations - -## Appendices - -### A. Technical Specifications -- Python 3.10+ -- Pulumi latest version -- AWS/Azure/GCP SDKs - -### B. Compliance Requirements -- NIST 800-53 -- FISMA Moderate -- ISO 27001 - -### C. Reference Architectures -- AWS Landing Zone -- Azure Landing Zone -- GCP Organization - -### D. Development Tools -- Poetry for dependency management -- Pyright for type checking -- pytest for testing diff --git a/docs/roadmaps/ROADMAP_Addendum.md b/docs/roadmaps/ROADMAP_Addendum.md deleted file mode 100644 index bc1359b..0000000 --- a/docs/roadmaps/ROADMAP_Addendum.md +++ /dev/null @@ -1,417 +0,0 @@ -# Konductor IaC Template Repository Refactor and Enhancement Roadmap - -> **Technical Blueprint Addendum** - -## Table of Contents - -1. [Executive Summary](#executive-summary) -2. [Technical Implementation Details](#technical-implementation-details) -3. [Module Development Guidelines](#module-development-guidelines) -4. [Infrastructure Components](#infrastructure-components) -5. [Security and Compliance](#security-and-compliance) -6. [Testing Strategy](#testing-strategy) -7. [Documentation Requirements](#documentation-requirements) -8. [Deployment Workflows](#deployment-workflows) -9. [Monitoring and Observability](#monitoring-and-observability) -10. [Risk Management](#risk-management) - -## Executive Summary - -This technical addendum provides detailed implementation guidance for the Konductor Platform Engineering initiative. It serves as a comprehensive reference for engineers at all levels, with specific focus on technical implementation details and best practices. - -### Audience-Specific Guidance - -#### Principal Engineers -- Architectural decision records -- System design considerations -- Performance optimization strategies -- Scalability patterns - -#### Senior Engineers -- Implementation patterns -- Code quality standards -- Testing strategies -- Module development guidelines - -#### Junior Engineers -- Setup procedures -- Development workflows -- Testing practices -- Documentation requirements - -## Technical Implementation Details - -### Core Architecture Components - -#### 1. Configuration Management - -```python -from typing import TypedDict, Optional, Dict, Any - -class BaseConfig(TypedDict): - """Base configuration for all modules.""" - enabled: bool - version: Optional[str] - parameters: Dict[str, Any] - tags: Dict[str, str] - -class ModuleConfig(BaseConfig): - """Extended configuration for specific modules.""" - dependencies: List[str] - providers: Dict[str, Any] - compliance: Dict[str, Any] - -# Example implementation -def load_module_config( - module_name: str, - config: Dict[str, Any] -) -> ModuleConfig: - """Load and validate module configuration.""" - base_config = get_base_config(module_name) - return merge_configs(base_config, config) -``` - -#### 2. Resource Management - -```python -class ResourceManager: - """Manages infrastructure resources.""" - - def __init__(self, config: ModuleConfig): - self.config = config - self.resources: List[Resource] = [] - - def create_resource( - self, - resource_type: str, - resource_config: Dict[str, Any] - ) -> Resource: - """Create and track a new resource.""" - resource = self._create_resource_internal( - resource_type, - resource_config - ) - self.resources.append(resource) - return resource -``` - -### Module Integration Framework - -#### 1. Standard Module Interface - -```python -from abc import ABC, abstractmethod -from typing import Optional, Dict, Any - -class ModuleInterface(ABC): - """Base interface for all modules.""" - - @abstractmethod - def deploy( - self, - config: ModuleConfig, - dependencies: Optional[List[Resource]] = None - ) -> Resource: - """Deploy module resources.""" - pass - - @abstractmethod - def validate(self) -> bool: - """Validate module configuration.""" - pass - - @abstractmethod - def cleanup(self) -> None: - """Cleanup module resources.""" - pass -``` - -## Module Development Guidelines - -### Module Structure - -``` -module_name/ -├── __init__.py -├── types.py -├── deploy.py -├── config.py -├── utils.py -├── tests/ -│ ├── __init__.py -│ ├── test_deploy.py -│ └── test_config.py -└── README.md -``` - -### Implementation Standards - -#### 1. Type Definitions - -```python -from typing import TypedDict, Optional - -class ModuleResourceConfig(TypedDict): - """Resource configuration for the module.""" - name: str - type: str - parameters: Dict[str, Any] - tags: Optional[Dict[str, str]] - -class ModuleDeploymentConfig(TypedDict): - """Deployment configuration for the module.""" - resources: List[ModuleResourceConfig] - dependencies: Optional[List[str]] - providers: Dict[str, Any] -``` - -#### 2. Deployment Logic - -```python -def deploy_module( - config: ModuleDeploymentConfig, - dependencies: Optional[List[Resource]] = None -) -> Resource: - """Deploy module resources with proper error handling.""" - try: - validate_config(config) - resources = create_resources(config) - return resources - except Exception as e: - handle_deployment_error(e) - raise -``` - -## Infrastructure Components - -### AWS Infrastructure - -#### 1. Network Architecture - -```python -class NetworkConfig(TypedDict): - """Network configuration structure.""" - vpc_cidr: str - subnet_cidrs: List[str] - availability_zones: List[str] - nat_gateways: int - -def create_network_stack( - config: NetworkConfig, - tags: Dict[str, str] -) -> NetworkStack: - """Create VPC and associated networking components.""" - vpc = create_vpc(config.vpc_cidr, tags) - subnets = create_subnets(vpc, config.subnet_cidrs) - return NetworkStack(vpc, subnets) -``` - -#### 2. Security Components - -```python -class SecurityConfig(TypedDict): - """Security configuration structure.""" - encryption_key_rotation: int - backup_retention_days: int - log_retention_days: int - -def configure_security( - config: SecurityConfig, - resources: List[Resource] -) -> None: - """Apply security configurations to resources.""" - for resource in resources: - apply_encryption(resource, config) - configure_backup_retention(resource, config) - setup_logging(resource, config) -``` - -## Security and Compliance - -### Compliance Framework - -#### 1. NIST Controls Implementation - -```python -class NistControl(TypedDict): - """NIST control implementation structure.""" - control_id: str - implementation: str - validation: str - monitoring: str - -def implement_nist_controls( - controls: List[NistControl], - resources: List[Resource] -) -> None: - """Implement NIST controls on resources.""" - for control in controls: - apply_control(control, resources) - validate_control(control, resources) -``` - -#### 2. Audit Logging - -```python -class AuditConfig(TypedDict): - """Audit logging configuration.""" - log_level: str - retention_period: int - encryption_enabled: bool - -def setup_audit_logging( - config: AuditConfig, - resources: List[Resource] -) -> None: - """Configure audit logging for resources.""" - logger = create_audit_logger(config) - for resource in resources: - enable_resource_logging(resource, logger) -``` - -## Testing Strategy - -### Automated Testing Framework - -#### 1. Unit Tests - -```python -import pytest -from typing import Generator - -@pytest.fixture -def module_config() -> Generator[ModuleConfig, None, None]: - """Provide test configuration.""" - config = create_test_config() - yield config - cleanup_test_config(config) - -def test_module_deployment( - module_config: ModuleConfig -) -> None: - """Test module deployment process.""" - result = deploy_module(module_config) - assert result.status == "SUCCESS" - validate_deployment(result) -``` - -#### 2. Integration Tests - -```python -def test_cross_module_integration( - module_a_config: ModuleConfig, - module_b_config: ModuleConfig -) -> None: - """Test integration between modules.""" - module_a = deploy_module(module_a_config) - module_b = deploy_module(module_b_config) - validate_integration(module_a, module_b) -``` - -## Documentation Requirements - -### Technical Documentation - -#### 1. Module Documentation Template - -```markdown -# Module Name - -## Overview -[Brief description of the module's purpose] - -## Configuration -```python -class ModuleConfig(TypedDict): - # Configuration structure - pass -``` - -## Usage Examples -[Code examples showing common use cases] - -## Implementation Details -[Technical details about the implementation] - -## Testing -[Instructions for testing the module] -``` - -## Deployment Workflows - -### CI/CD Pipeline - -#### 1. Build and Test - -```yaml -# GitHub Actions workflow -name: Build and Test -on: [push, pull_request] - -jobs: - build: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: '3.8' - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install poetry - poetry install - - name: Run tests - run: poetry run pytest -``` - -## Monitoring and Observability - -### Telemetry Implementation - -#### 1. Metrics Collection - -```python -class MetricsConfig(TypedDict): - """Metrics collection configuration.""" - namespace: str - dimensions: Dict[str, str] - period: int - -def setup_metrics( - config: MetricsConfig, - resources: List[Resource] -) -> None: - """Configure metrics collection for resources.""" - metrics_client = create_metrics_client(config) - for resource in resources: - enable_resource_metrics(resource, metrics_client) -``` - -## Risk Management - -### Risk Mitigation Strategies - -#### 1. Deployment Safeguards - -```python -class DeploymentSafeguards(TypedDict): - """Deployment safety configuration.""" - rollback_enabled: bool - validation_timeout: int - max_retry_attempts: int - -def apply_deployment_safeguards( - config: DeploymentSafeguards, - deployment: Deployment -) -> None: - """Apply safety measures to deployment.""" - configure_rollback(deployment - -, config) - set_validation_checks(deployment, config) - configure_retry_policy(deployment, config) -``` - -This technical addendum provides detailed implementation guidance while maintaining alignment with the main roadmap. It serves as a comprehensive reference for engineers at all levels, ensuring consistent implementation across the platform.