Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline)
Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline)¶
- Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming
- Message Brokers
- Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions
- Debezium open source distributed platform for Change Data Capture (CDC) software design pattern
- Red Hat Integration service registry and Apicurio
- Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)
- Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)
- eBooks
- Related
Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming¶
Message Brokers¶
- Apache ActiveMQ
- Dzone: Introduction to Message Brokers. Part 1: Apache Kafka vs. RabbitMQ
- Dzone: Introduction to Message Brokers. Part 2: ActiveMQ vs. Redis Pub/Sub
ActiveMQ message broker¶
- ActiveMQ 5.x “classic”
- ActiveMQ Artemis Apache ActiveMQ is a subproject of Apache ActiveMQ. It has been donated to the Apache Software Foundation in 2015. There were lots of changes in project names in the past. The Artemis project first started as JBoss Messaging and got renamed to HornetQ in August 2009.
- Apache Artemis JMeter Running the ActiveMQ Artemis JMeter Performance Testing Examples.
RabbitMQ message broker¶
- K8s prevent queue worker Pod from being killed during deployment How to prevent a Kubernetes (like RabbitMQ) queue worker Pod from being killed during deployment while handling a message?
- medium.com: RabbitMQ vs. Kafka An architect’s dilemma
Redis message broker¶
Apache Camel message broker¶
Apache Camel K¶
- Apache Camel K is a lightweight cloud-integration platform that runs natively on Kubernetes. Based on the famous Apache Camel, Camel K is designed and optimized for serverless and microservices architectures.
- developers.redhat.com: Six reasons to love Camel K
KubeMQ message broker¶
- KubeMQ.io: Kubernetes Native Message Queue Broker
- devops.com: Best of 2019: Implementing Message Queue in Kubernetes
- kubemq.io: Kafka VS KubeMQ 🌟
Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions¶
- Wikipedia: Cloud Based Integration (iPaaS)
- Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
- Integration platform as a service (iPaaS) is a set of automated tools for connecting software applications that are deployed in different environments. iPaaS is often used by large business-to-business (B2B) enterprises that need to integrate on-premises applications and data with cloud applications and data.
- blog.axway.com: What is iPaaS?
- ibm.com: iPaaS (Integration-Platform-as-a-Service): iPaaS is a cloud-based solution that simplifies application integration across on-premises and cloud environments, to help you accelerate innovation and lower your integration and operations costs.
Red Hat Fuse and Red Hat Fuse Online¶
Syndesis open source integration platform¶
- Syndesis open source integration platform (OpenSource Project for Red Hat Fuse Online)
- developers.redhat.com: Low-code microservices orchestration with Syndesis
Debezium open source distributed platform for Change Data Capture (CDC) software design pattern¶
- Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
- Debezium: Stream changes from your database
- developers.redhat.com: Decoupling microservices with Apache Camel and Debezium
- A good explanation of how to avoid distributed transactions using outbox pattern: Transaction Log Tailing With Debezium
- developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
Red Hat Integration service registry and Apicurio¶
- Red Hat Integration service registry
- Apicurio Registry An API/Schema registry - stores APIs and Schemas.
Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)¶
- Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources.
Apache Kafka¶
- Apache Kafka
- developers.redhat.com: how easy to deploy and configure a Kafka Connect on Kubernetes through strimziio operator and use secrets
- developers.redhat.com: Using secrets in Kafka Connect configuration
- developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
- Awesome Kafka
Banzai Cloud Supertubes (Cloud Native Kafka implementation)¶
Apache Flink¶
- Apache Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
- How to set up Apache Flink on Kubernetes for real time data processing
Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)¶
- Red Hat AMQ overview
- Red Hat AMQ = AMQ Broker (Apache ActiveMQ Artemis) + AMQ Streams (Apache Kafka)
Red Hat AMQ Broker (ActiveMQ Artemis)¶
- Apache ActiveMQ Artemis broker
- developers.redhat.com: JDBC Master-Slave Persistence setup with Activemq using Postgresql database
Red Hat AMQ Streams¶
- Understanding Red Hat AMQ Streams components for OpenShift and Kubernetes 🌟
- Red Hat AMQ streams (kafka): Simplify Apache Kafka on Red Hat OpenShift
- Set up Red Hat AMQ Streams custom certificates on OpenShift
| Product | Also Known As | Components | URL |
|---|---|---|---|
| Red Hat AMQ 6 | JBoss AMQ 6 | Apache ActiveMQ | Ref |
| Red Hat AMQ 7 | JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite | AMQ Broker + AMQ Streams | Ref |
| Red Hat AMQ 7 | JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite | JBoss AMQ 7 (Broker) + Apache Kafka | Ref |
| Red Hat AMQ 7 | JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite | Apache ActiveMQ Artemis + Apache Kafka | Ref |