DataJourney
lastmod 2024-12-26created 2024-12-261 edits
Design-First open-source data management toolkit
About
- Provides a structured approach that leverages the strengths of open-source technology to help organizations manage and use data effectively
- Emphasizes scalability and reproducibility, and guides you through the essential steps needed to build data workflows
- Systematic support covering goal setting, tool selection, and workflow testing and customization
- Flexible, modular design that can be adapted to user needs
Design philosophy: Layer structure
- PO (Foundation): Serves as a static home, such as GitHub
- P1 (Tools): Various tools powered by open source
- P2 (Maintenance and monitoring): Environment and automation management (Pixi and GHA)
- P3 (Abstraction): CLI/task manager layer for user interaction (Pixi)
Currently supported workflows
- Implementing Python packaging framework design principles
- Setting up GitHub Actions
- Configuring Vale.sh at the PR level
- Setting up code linting/formatting with pre-commit hooks
- Environment management with Pixi
- Reading online data sources with Intake
- Building a sample pipeline with Dagster
- Building dashboards with Holoviews + Panel
- Exploratory data analysis (EDA) with Mito
- Building a Flask-based web UI
- Extending and restructuring the web UI with FastHTML
- Performing data analysis with GitHub AI models (GitHub AI models Beta)