Data and ML Stack Tools by Category

Reference architectures

Three ways to wire the tools above into a working stack. All of them have the same three layers, ingestion, processing and storage, plus whatever consumes the result.

1. Batch lakehouse

Nightly and hourly ELT. The default when latency is measured in hours.

  flowchart TB
  subgraph ing["Ingestion"]
    airbyte["Airbyte"]
    debezium["Debezium + Kafka Connect"]
  end
  subgraph proc["Processing"]
    spark["Apache Spark"]
    dbt["dbt"]
    ge["Great Expectations"]
  end
  subgraph sto["Storage"]
    iceberg["Apache Iceberg"]
    s3["MinIO / S3"]
    polaris["Apache Polaris"]
  end
  subgraph out["Consumption"]
    trino["Trino"]
    superset["Apache Superset"]
  end

  airbyte --> spark
  debezium --> spark
  spark --> dbt --> ge --> iceberg
  s3 -.- iceberg
  polaris -.- iceberg
  iceberg --> trino --> superset

Apache Airflow sits outside the picture, triggering every arrow above.

2. Streaming

Seconds instead of hours. Two branches out of one job: the table for history, the OLAP store for the dashboard.

  flowchart TB
  subgraph ing["Ingestion"]
    debezium["Debezium"]
    kafka["Apache Kafka"]
  end
  subgraph proc["Processing"]
    flink["Apache Flink"]
  end
  subgraph sto["Storage"]
    paimon["Apache Paimon on S3"]
    clickhouse["ClickHouse"]
  end
  subgraph out["Consumption"]
    trino["Trino"]
    grafana["Grafana"]
  end

  debezium --> kafka --> flink
  flink --> paimon --> trino
  flink --> clickhouse --> grafana

Airflow is still needed here, not to schedule the job but to run the compaction and snapshot expiration the table needs because the job never stops writing.

3. ML and inference

Features written by the same pipeline that feeds training, and a model served out of a registry.

  flowchart TB
  subgraph ing["Ingestion"]
    kafka["Apache Kafka"]
  end
  subgraph proc["Processing"]
    flink["Apache Flink"]
    spark["Apache Spark"]
  end
  subgraph sto["Storage"]
    feast["Feast"]
    delta["Delta Lake on S3"]
    qdrant["Qdrant"]
  end
  subgraph train["Training"]
    ray["Ray"]
    mlflow["MLflow Registry"]
  end
  subgraph serve["Serving"]
    kserve["KServe"]
    vllm["vLLM"]
  end

  kafka --> flink
  kafka --> spark
  flink --> feast
  flink --> qdrant
  spark --> delta
  delta --> ray --> mlflow --> kserve
  feast --> kserve
  qdrant --> vllm

Official repositories

• Airbyte: Connector-based ELT for loading data into a warehouse or lake.
• Amundsen: Data discovery and metadata search for analysts.
• Apache Airflow: Schedules and monitors pipelines written as Python DAGs.
• Apache Arrow: In-memory columnar format shared between engines and languages.
• Apache Atlas: Metadata and lineage governance for the Hadoop ecosystem.
• Apache Avro: Row-oriented serialization format that carries its schema.
• Apache Beam: One pipeline model that runs on several batch and streaming engines.
• Apache Doris: MPP analytical database aimed at real-time reporting.
• Apache Flink: Stateful stream processing with event time and exactly-once.
• Apache Hudi: Lake table format with upserts and incremental reads.
• Apache Iceberg: Open table format with ACID commits, schema evolution and time travel.
• Apache Kafka: Distributed log for publishing and consuming event streams.
• Apache NiFi: Flow-based routing and transformation with a visual editor.
• Apache ORC: Columnar file format built for large Hive-style reads.
• Apache Paimon: Lake table format designed to be written continuously by Flink.
• Apache Parquet: The columnar file format most of the lake is stored in.
• Apache Polaris: Open catalog so several engines see the same Iceberg tables.
• Apache Pulsar: Messaging and streaming with storage separated from serving.
• Apache Spark: General engine for batch, SQL and ML over large datasets.
• Apache Superset: Web BI with dashboards and SQL exploration.
• Argo Workflows: Runs container-native workflows as Kubernetes objects.
• BentoML: Packages a model and its dependencies into a service.
• ClickHouse: Column store for fast analytical queries at scale.
• Dagster: Orchestrator built around data assets rather than tasks.
• Dataform: SQL transformation workflows with dependency management.
• DataHub: Metadata platform for catalog, lineage and ownership.
• dbt: Turns SQL selects into tested, documented and versioned models.
• Debezium: Reads database change logs and publishes them as events.
• Deequ: Data quality checks expressed as Spark jobs.
• Delta Lake: Transactional storage layer over Parquet files.
• Dremio: Query engine and semantic layer over lake storage.
• DuckDB: Analytical database that runs inside your process, no server.
• Feast: Feature store serving the same features to training and inference.
• Flyte: Typed, versioned workflows for data and ML on Kubernetes.
• Grafana: Dashboards over metrics, logs and traces.
• Great Expectations: Declares expectations about data and validates them.
• Hive Metastore: The table and partition catalog many engines still read from.
• Hopsworks: Feature store and ML platform.
• Jupyter: Notebooks for interactive analysis.
• KServe: Serves models on Kubernetes with autoscaling and canaries.
• Kubeflow: ML platform assembled from Kubernetes components.
• Kubeflow Pipelines: Composes and runs ML pipelines as containers.
• LightGBM: Gradient boosting tuned for speed on large tabular data.
• Marquez: Reference server that collects and stores OpenLineage events.
• Metabase: BI that lets non-engineers ask questions of the database.
• Metaflow: Python workflows for data science, from Netflix.
• Milvus: Vector database for similarity search at scale.
• MinIO: S3-compatible object storage you run yourself.
• MLflow: Tracks experiments, packages models and holds the registry.
• NumPy: Arrays and numerical computing, the base of the Python stack.
• NVIDIA Triton: Inference server for models on GPU and CPU.
• OpenLineage: Open standard for pipelines to report what they read and wrote.
• OpenMetadata: Catalog, lineage, quality and governance in one place.
• OpenTelemetry: Vendor-neutral standard for traces, metrics and logs.
• Pandas: DataFrames for data manipulation in Python.
• pgvector: Vector types and indexes inside PostgreSQL.
• Polars: Fast DataFrames written in Rust, with a lazy query engine.
• Prefect: Orchestration for Python workflows, built for dynamic runs.
• Presto: Distributed SQL engine that queries data where it lives.
• Project Nessie: Git-like branches and commits for lake tables.
• Prometheus: Time series database and alerting for metrics.
• Python: The language most of this stack is written and driven in.
• PyTorch: Deep learning framework with dynamic graphs.
• Qdrant: Vector search engine with filtering, written in Rust.
• R: Language and environment for statistics.
• Ray: Distributed Python runtime for training, tuning and serving.
• Redpanda: Kafka-compatible broker with no JVM and no ZooKeeper.
• Scikit-learn: Classical machine learning algorithms in Python.
• SciPy: Scientific computing built on NumPy.
• Seldon: Deploys and monitors models on Kubernetes.
• Soda: Data quality checks written as readable rules.
• SQLMesh: Transformations with column-level lineage and virtual environments.
• StarRocks: MPP OLAP database for sub-second queries over the lake.
• TensorFlow: Deep learning framework with a production toolchain.
• Trino: Distributed SQL that federates queries across many sources.
• vLLM: High-throughput serving for large language models.
• Weaviate: Vector database with built-in embedding modules.
• XGBoost: Gradient boosting library, the workhorse for tabular data.

Kafka Connect ships inside apache/kafka, dbt Tests inside dbt-core and Ray Serve inside ray-project/ray, so they share the repositories above.

No public repository, these are proprietary or managed services: Power BI, Tableau, Fivetran, Amazon S3, Google Cloud Storage, Azure Data Lake Storage, AWS Glue Catalog, Databricks, Starburst, Snowflake, BigQuery, Redshift, Tecton, Pinecone, Weights & Biases, Neptune, Comet, Vertex AI, SageMaker and Azure ML.