Portfolio Architecture Examples – Data Engineering Collection
For a few years now we’ve been working on a project we have named Portfolio Architectures. These are based on selecting a specific use case we are seeing used in the real world by customers and then finding implementations of that case using three or more products from the Red Hat portfolio.
This basic premise is used as the foundation, but many aspects of open source are included in both the process and the final product we have defined. There is a community, where we share the initial project kickoff with a group of architects and use their initial feedback from the start. We also present the architecture product we’ve created right at the end before we publish to ensure usability by architects in the field. The final publish product includes some internal only Content around the customer projects researched, but most of the Content is open and freely available through various open source channels.
This article is sharing an overview of the product we’ve developed, what’s available to you today in our architecture center, and concludes by sharing a collection of architectures we’ve published.
The basis of a portfolio architecture is a use case, two to three actual implementations that can be researched, and includes the use of a minimum of three products. This is the ideal foundation for a project to start, but we encountered a problem with use cases containing emerging technologies or emerging domains in the market. To account for these we’ve chosen to note the fact that these are opinionated architectures based on internal reference architectures.
The product has been defined as complete for publishing when it contains the following Content:
- Short use case definition
- Diagrams – logical, schematic (physical), and detail diagrams
- Public slide deck containing the use case story and architecture diagrams
- Internal slide deck containing both the public deck Content and the confidential customer research
- Video (short) explanation of the architecture
- Either a technical brief document or one or more articles covering the solution architecture
Note that the items in italics are all available to anyone in the Red Hat Portfolio Architecture Center or in the Portfolio Architecture Examples repository.
Tooling and workshops
The progress towards our products required a good idea of how we wanted to diagram our architectures. We chose to keep them very generic and simple in style to facilitate all levels of conversation around a particular use case without getting bogged down in notational discussions.
A simple three level design for our architectures was captured by using logical, schematic, and detail diagrams. All of these have been integrated in open source tooling with pre-defined templates and icons for easily getting started. Furthermore, we’ve developed a tooling workshop to quickly ramp up on the design methods and tooling we’ve made available. It’s called Designing Your Best Architectural Diagrams, has been featured in several conferences around the world.
Data engineering collection
The collection featured today is centred around architectures leveraging data engineering concepts and tooling. There are currently eight architectures in this collection and we’ll provide a short overview of each, leaving the in depth exploration as an exercise for the reader.
In each of these architecture overviews you’ll find a table of Contents outlining the technologies used, several example schematic diagrams with descriptions, and a link in the last section to open the diagrams directly into the online tooling in your browser.
This architecture covers the use case of providing a consistent infrastructure experience from cloud to edge and enabling modern containerized applications at edge.
(Note: this project is a new architecture and currently in progress, so sharing one of the schematic architecture diagrams and you can monitor this project for updates as it progresses to completion.)
This use case is bringing cloud like capabilities to the edge locations.
This architecture covers the use case around data center to edge where the energy (utility) infrastructure companies operate across vast geographical area that connects the upstream drilling operations with downstream fuel processing and delivery to customers. These companies need to monitor the condition of pipeline and other infrastructure for operational safety and optimization.
The use case is bringing computing closer to the edge by monitoring for potential issues with gas pipelines (edge).
The manufacturing industry has consistently used technology to fuel innovation, production optimization and operations. Now, with the combination of edge computing and AI/ML, manufacturers can benefit from bringing processing power closer to data. This helps actions be taken faster on things like errors and predictive maintenance.
The use case is for boosting manufacturing efficiency and product quality with artificial intelligence and machine learning out to the edge.
This architecture covers edge medical diagnosis in the healthcare industry. It Accelerates medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities.
The use case is accelerating medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities.
Intelligent Data as a Service (iDaaS)
Intelligent DaaS (Data as a Service) is about building and delivery of systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerisation in healthcare. Feel free to explore this portfolio architecture by clicking on the diagram below.
The use case is Intelligent Data as a Service (iDaaS) is about building and delivery of systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerisation in healthcare.
An offering of (near) real-time payments lets businesses, consumers, and even governments send and accept funds that provide both availability to the recipient and instant confirmation to the sender. Enabling real-time – or at least faster – payments that improve the speed of online payment experiences to customers has the potential to give banks a greater opportunity to win, serve, and retain their customers. By building solutions that capture real-time payment business, banks also can drive higher payment volumes, ideally at lower costs as well as engage new customer segments.
The use case examines financial institutions enabling customers with fast, easy to use, and safe payment services available anytime, anywhere.
Retail is the process of selling consumer goods or services to customers through multiple channels of distribution to earn a profit. A data framework refers to the process of managing enterprise retail data. The framework or system sets the guidelines and rules of engagement for business and management activities, especially those that deal with or result in the creation and manipulation of data.
The use case is creating a framework for access to retail data from customers, stock, stores, and staff across multiple internal teams.
This Portfolio Architecture targets energy providers in North America that need to be compliant with NERC regulations and in order to achieve this they decide to modernise the interfaces between their business applications and their SCADA systems also for a better consumption of information that can be used in combination with AI/ML and decision management tools to better address customer needs.
The use case is providing interfaces with SCADA systems that are compliant with NERC regulations, creating different layers of API gateways to protect business service depending on the network zones.
If you are interested in more architecture solutions like these, feel free to explore the Portfolio Architecture Examples repository. More architecture collections include:
- Application development
- Data engineering
Published on Java Code Geeks with permission by Eric Schabell, partner at our JCG program. See the original article here: Portfolio Architecture Examples – Data Engineering Collection
Opinions expressed by Java Code Geeks contributors are their own.