Data Science & ML Pipeline Architecture Pack — 50 Prompts

From Jupyter Notebook to Production ML — The Complete Pipeline

90% of ML models never make it to production. The gap isn't model accuracy — it's engineering. These 50 prompts bridge that gap, giving data scientists and ML engineers systematic approaches for every stage from data quality assessment to production monitoring.

Prompts use chain-of-thought for complex statistical reasoning, few-shot examples with real dataset scenarios, and tree-of-thought for model selection trade-offs. Variables like {{dataset_description}}, {{business_metric}}, {{latency_requirement}}, and {{team_expertise}} ensure practical, contextual recommendations.

What's Inside — 50 Expert Prompts

• Feature Engineering Strategist — Analyzes {{raw_features}} and generates engineered features with rationale, interaction terms, temporal features, and encoding strategies for {{model_type}}.
• Model Selection Advisor — Compares algorithms for {{problem_type}} (classification, regression, ranking, clustering) with accuracy, interpretability, latency, and data requirement trade-offs.
• Hyperparameter Tuning Strategy — Designs search strategy for {{model}} using Bayesian optimization, early stopping, and resource-efficient approaches for {{compute_budget}}.
• Experiment Tracking Framework — Sets up MLflow/Weights&Biases experiment tracking for {{project}} with metric logging, artifact management, and reproducibility guarantees.
• MLOps Deployment Pipeline Designer — Creates production ML pipeline for {{model}} with feature store, model registry, A/B testing, shadow deployment, and automated rollback.
• Model Monitoring Dashboard — Designs monitoring for {{production_model}} covering prediction drift, feature drift, data quality, latency, and business metric correlation.
• A/B Test Design Calculator — Designs experiments for {{hypothesis}} with sample size calculation, MDE, power analysis, duration estimation, and sequential testing options.
• Statistical Analysis Framework — Conducts statistical analysis for {{research_question}} with appropriate test selection, assumption checking, effect size, and confidence intervals.
• Data Quality Assessment — Audits {{dataset}} for completeness, consistency, accuracy, timeliness, and uniqueness with automated quality scoring and remediation recommendations.
• Feature Store Designer — Architects feature store for {{organization}} with online/offline stores, feature versioning, backfill strategies, and point-in-time correctness.
• Time Series Forecasting Pipeline — Designs forecasting for {{metric}} with seasonality decomposition, model selection (ARIMA, Prophet, neural), and ensemble strategies.
• NLP Pipeline Architect — Builds text processing pipeline for {{task}} (classification, NER, summarization, sentiment) with preprocessing, embedding selection, and evaluation.
• Recommender System Designer — Creates recommendation engine for {{product_type}} using collaborative, content-based, or hybrid approaches with cold-start handling.
• Data Pipeline Orchestrator — Designs ETL/ELT pipeline for {{data_sources}} with Airflow/Dagster DAGs, incremental processing, and data validation gates.
• Model Interpretability Reporter — Generates interpretability analysis for {{model}} using SHAP values, feature importance, partial dependence plots, and counterfactual explanations.

Each Prompt Includes

• {{placeholder}} variables for dataset, model type, business context, and infrastructure constraints
• Expected output: code snippets, architecture diagrams, statistical reports, or configuration files
• Chain-of-thought reasoning for statistical decisions and tree-of-thought for model selection
• Anti-patterns: data leakage, training-serving skew, metric gaming, and p-hacking warnings

Who This Is For

• Data scientists who want to ship models to production, not just notebooks
• ML engineers building reliable, monitored ML infrastructure
• Analytics managers designing experimentation and statistical frameworks
• CTOs evaluating ML feasibility and infrastructure requirements for their products

What Makes This Different

• Covers the ENTIRE ML lifecycle — not just model training but data quality, feature engineering, deployment, and monitoring
• Production-focused — every prompt considers latency, reliability, and maintainability alongside accuracy
• Includes statistical rigor — proper A/B testing, confidence intervals, and effect size analysis

Works With

ChatGPT (GPT-4+), Claude (3.5+), Gemini Pro. Best with ChatGPT for code generation, Claude for statistical reasoning.