GitHub - spring-attic/spring-cloud-dataflow: A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes · GitHub
Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert This repository was archived by the owner on May 14, 2025. It is now read-only. Branches Tags Open more actions menu Latest commit History 5,144 Commits 5,144 Commits Folders and files Name Name Last commit message Last commit date .github .github .mvn .mvn .run .run .vscode .vscode lib lib models models spring-cloud-common-security-config spring-cloud-common-security-config spring-cloud-dataflow-audit spring-cloud-dataflow-audit spring-cloud-dataflow-autoconfigure spring-cloud-dataflow-autoconfigure spring-cloud-dataflow-build spring-cloud-dataflow-build spring-cloud-dataflow-classic-docs spring-cloud-dataflow-classic-docs spring-cloud-dataflow-common spring-cloud-dataflow-common spring-cloud-dataflow-completion spring-cloud-dataflow-completion spring-cloud-dataflow-composed-task-runner spring-cloud-dataflow-composed-task-runner spring-cloud-dataflow-configuration-metadata spring-cloud-dataflow-configuration-metadata spring-cloud-dataflow-container-registry spring-cloud-dataflow-container-registry spring-cloud-dataflow-core-dsl spring-cloud-dataflow-core-dsl spring-cloud-dataflow-core spring-cloud-dataflow-core spring-cloud-dataflow-dependencies spring-cloud-dataflow-dependencies spring-cloud-dataflow-docs spring-cloud-dataflow-docs spring-cloud-dataflow-package spring-cloud-dataflow-package spring-cloud-dataflow-parent spring-cloud-dataflow-parent spring-cloud-dataflow-platform-cloudfoundry spring-cloud-dataflow-platform-cloudfoundry spring-cloud-dataflow-platform-kubernetes spring-cloud-dataflow-platform-kubernetes spring-cloud-dataflow-registry spring-cloud-dataflow-registry spring-cloud-dataflow-rest-client spring-cloud-dataflow-rest-client spring-cloud-dataflow-rest-resource spring-cloud-dataflow-rest-resource spring-cloud-dataflow-server-core spring-cloud-dataflow-server-core spring-cloud-dataflow-server spring-cloud-dataflow-server spring-cloud-dataflow-shell-core spring-cloud-dataflow-shell-core spring-cloud-dataflow-shell spring-cloud-dataflow-shell spring-cloud-dataflow-single-step-batch-job spring-cloud-dataflow-single-step-batch-job spring-cloud-dataflow-tasklauncher spring-cloud-dataflow-tasklauncher spring-cloud-dataflow-test spring-cloud-dataflow-test spring-cloud-skipper spring-cloud-skipper spring-cloud-starter-dataflow-server spring-cloud-starter-dataflow-server spring-cloud-starter-dataflow-ui spring-cloud-starter-dataflow-ui src src .editorconfig .editorconfig .gitignore .gitignore .sdkmanrc .sdkmanrc .settings.xml .settings.xml .springjavaformatconfig .springjavaformatconfig .trivyignore .trivyignore CONTRIBUTING.adoc CONTRIBUTING.adoc LICENSE LICENSE README.md README.md SECURITY.md SECURITY.md build-carvel-package.sh build-carvel-package.sh build-containers.sh build-containers.sh mvnw mvnw mvnw.cmd mvnw.cmd pom.xml pom.xml run-integration-test.sh run-integration-test.sh run-integration-tests.sh run-integration-tests.sh trivy.yaml trivy.yaml whats-new.adoc whats-new.adoc Repository files navigation Spring Cloud Data Flow is no longer maintained as an open-source project by Broadcom, Inc.
For information about extended support or commercial options for Spring Cloud Data Flow, please read the official blog post here . Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes. Data processing pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks. This makes Spring Cloud Data Flow ideal for a range of data processing use cases, from import/export to event streaming and predictive analytics.
Components Architecture : The Spring Cloud Data Flow Server is a Spring Boot application that provides RESTful API and REST clients (Shell, Dashboard, Java DSL). A single Spring Cloud Data Flow installation can support orchestrating the deployment of streams and tasks to Local, Cloud Foundry, and Kubernetes. Familiarize yourself with the Spring Cloud Data Flow architecture and feature capabilities . Deployer SPI : A Service Provider Interface (SPI) is defined in the Spring Cloud Deployer project.
The Deployer SPI provides an abstraction layer for deploying the apps for a given streaming or batch data pipeline and managing the application lifecycle. Spring Cloud Deployer Implementations: Local Cloud Foundry Kubernetes Domain Model : The Spring Cloud Data Flow domain module includes the concept of a stream that is a composition of Spring Cloud Stream applications in a linear data pipeline from a source to a sink , optionally including processor application(s) in between.
The domain also includes the concept of a task , which may be any process that does not run indefinitely, including Spring Batch jobs. Application Registry : The App Registry maintains the metadata of the catalog of reusable applications. For example, if relying on Maven coordinates, an application URI would be of the format: maven://<groupId>:<artifactId>:<version> . Shell/CLI : The Shell connects to the Spring Cloud Data Flow Server's REST API and supports a DSL that simplifies the process of defining a stream or task and managing its lifecycle.
Building Clone the repo and type $ ./mvnw -s .settings.xml clean install Looking for more information? Follow this link . Building on Windows When using Git on Windows to check out the project, it is important to handle line-endings correctly during checkouts. By default Git will change the line-endings during checkout to CRLF . This is, however, not desired for Spring Cloud Data Flow as this may lead to test failures under Windows.
Therefore, please ensure that you set Git property core.autocrlf to false , e.g. using: $ git config core.autocrlf false . For more information please refer to the Git documentation, Formatting and Whitespace . Running Locally w/ Oracle By default, the Dataflow server jar does not include the Oracle database driver dependency. If you want to use Oracle for development/testing when running locally, you can specify the local-dev-oracle Maven profile when building.
The following command will include the Oracle driver dependency in the jar: $ ./mvnw -s .settings.xml clean package -Plocal-dev-oracle You can follow the steps in the Oracle on Mac ARM64 Wiki to run Oracle XE locally in Docker with Dataflow pointing at it. NOTE: If you are not running Mac ARM64 just skip the steps related to Homebrew and Colima Running Locally w/ Microsoft SQL Server By default, the Dataflow server jar does not include the MSSQL database driver dependency.
If you want to use MSSQL for development/testing when running locally, you can specify the local-dev-mssql Maven profile when building. The following command will include the MSSQL driver dependency in the jar: $ ./mvnw -s .settings.xml clean package -Plocal-dev-mssql You can follow the steps in the MSSQL on Mac ARM64 Wiki to run MSSQL locally in Docker with Dataflow pointing at it. NOTE: If you are not running Mac ARM64 just skip the steps related to Homebrew and Colima Running Locally w/ IBM DB2 By default, the Dataflow server jar does not include the DB2 database driver dependency.
If you want to use DB2 for development/testing when running locally, you can specify the local-dev-db2 Maven profile when building. The following command will include the DB2 driver dependency in the jar: $ ./mvnw -s .settings.xml clean package -Plocal-dev-db2 You can follow the steps in the DB2 on Mac ARM64 Wiki to run DB2 locally in Docker with Dataflow pointing at it. NOTE: If you are not running Mac ARM64 just skip the steps related to Homebrew and Colima Contributing We welcome contributions!
See the CONTRIBUTING guide for details. Code formatting guidelines The directory ./src/eclipse has tw
The rest of this page is on the live site.