README.md 16 KB
Newer Older
1
# Qorus Integration Engine® Building Blocks
Ondrej Sejvl's avatar
Ondrej Sejvl committed
2

3
4
5
6
7
8
This repository contains building blocks or components for Qorus Integration Engine®

building blocks in any supported language can be used by code in other languages, so just because you are working with a code in Python, you don't have to restrict yourself to Python-based building blocks, Java, Python, and Qore-based building blocks can be mixed in any solution.

Building blocks are meant to provide reusable elements to solve technical challenges in a generic way to reduce development effort - ideally to eliminate development and transform the delivery of complex functionality to simply configuration.

David Nichols's avatar
David Nichols committed
9
10
11
**NOTE: The building blocks here are made available under the Apache license 2.0:**
>
> Copyright 2021 Qore Technologies, s.r.o.
12
>
David Nichols's avatar
David Nichols committed
13
> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.  You may obtain a copy of the License at
14
>
David Nichols's avatar
David Nichols committed
15
>     http://www.apache.org/licenses/LICENSE-2.0
16
>
David Nichols's avatar
David Nichols committed
17
18
19
20
> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.

Contact Qore Technologies (info@qoretechnologies.com) if you need support for these building blocks.

21
22
23
24
25
26
27
28
## Installation

Qorus must be installed, and the command-line environment must be set up for Qorus as well; also GNU make should be available in the `PATH`.

After cloning the repository, type `make load-bb`

## Building Blocks Index

David Nichols's avatar
David Nichols committed
29
30
31
32
33
34
35
36
37
38
39
40
**NOTE: Some building blocks have dependencies in other building blocks, modules, and/or data types; when deploying
single building blocks, ensure that all dependencies are deployed at the same time.  Keep in mind that modules often
require system option changes in `$OMQ_DIR/etc/options` (ex: `qorus.dataprovider-modules`) which require a Qorus
restart to take effect.**

**NOTE: Qorus must be restarted after installing the full building block release or any release with module dependencies
where the `qorus.dataprovider-module` option was updated for the first time in order for system option changes to take
effect.**

**NOTE: Qorus types must be deployed first in Qorus before they can be used in the IDE in mappers or workflow static
data types, etc.**

41
42
43
**NOTE: If step building blocks are subclassed, make sure and either remove the `primary()` or `validation()` method
or override it and make an explicit call to the parent class method in the child class

44
45
The following building blocks are available:

46
47
48
49
50
51
### AI / NLU / ONDEWO Building Blocks

|Building Block|Description|
|---|---|
|[BBM_OndewoBpiServer](./ondewo/BBM_OndewoBpiServer-1.0.qclass.yaml)|ONDEWO Business Process Integration server building block with event-based connectors for intent handling
|[BBM_OndewoIntentHandler](./ondewo/BBM_OndewoIntentHandler-1.0.qclass.yaml)|ONDEWO intent handler helper class
52
53
|[BBM_OndewoNluClient](./ondewo/BBM_OndewoNluClient-1.0.qclass.yaml)|ONDEWO Natural Language Understanding client class
|[BBM_OndewoVtsiClient](./ondewo/BBM_OndewoVtsiClient-1.0.qclass.yaml)|ONDEWO VTSI client class
54

55
56
57
58
59
60
### Data Provider / Mapper Building Blocks

|Building Block|Description|
|---|---|
|[BBM_AutoMapper](./generic/dataprovider/BBM_AutoMapper-v1.0.qclass.yaml)|base building block for running a mapper in autonomous mode; includes an input/output connector
|[BBM_AutoMapperRecord](./generic/dataprovider/BBM_AutoMapperRecord-1.0.qclass.yaml)|base building block for running a mapper in autonomous mode with request/record-based recovery logic
61
|[BBM_AutoMapperRecordStep](./step/dataprovider/BBM_AutoMapperRecordStep-1.0.qclass.yaml)|step building block to map record-based data from an output provider to an input provider with config-based error recovery support; runs a mapper in autonomous mode
62
|[BBM_AutoMapperRequest](./generic/dataprovider/BBM_AutoMapperRequest-1.0.qclass.yaml)|base building block for running a mapper in autonomous mode with request/response-based recovery logic
63
|[BBM_CsvReadDataProvider](./generic/dataprovider/BBM_CsvReadDataProvider-1.0.qclass.yaml)|base building block for processing CSV data as input; implements an output connector
64
|[BBM_DataProviderRecordCreate](./generic/dataprovider/BBM_DataProviderRecordCreate-1.0.qclass.yaml)|building block for record-based data providers for creating records, includes an input/output connector
65
66
67
68
69
70
71
72
73
|[BBM_DataProviderRecordUpdate](./generic/dataprovider/BBM_DataProviderRecordUpdate-1.0.qclass.yaml)|building block for record-based data providers for updating records, includes an input/output connector
|[BBM_DataProviderRecordCreateBase](./generic/dataprovider/BBM_DataProviderRecordCreateBase-1.0.qclass.yaml)|base building block for record-based data providers for record creation
|[BBM_DataProviderRecordUpdateBase](./generic/dataprovider/BBM_DataProviderRecordUpdateBase-1.0.qclass.yaml)|base base class for record-based data providers for creating records
|[BBM_DataProviderRequest](./generic/dataprovider/BBM_DataProviderRequest-1.0.qclass.yaml)|building block for request-based data providers; the request message body is created from config, includes an input/output connector
|[BBM_DataProviderRequestBase](./generic/dataprovider/BBM_DataProviderRequestBase-1.0.qclass.yaml)|base class for using the data-provider request API to make requests
|[BBM_DataProviderRequestWithRecovery](./generic/dataprovider/BBM_DataProviderRequestWithRecovery-1.0.qclass.yaml)|building block for request-based data providers with recovery logic; supports recovery with a single value, includes connectors for using in workflow steps for fault-tolerant operation
|[BBM_DataProviderRequestWithRecoveryStep](./step/dataprovider/BBM_DataProviderRequestWithRecoveryStep-1.0.qstep.yaml)|step building block for request-reply data providers with config-based error recovery support
|[BBM_DataProviderSearch](./generic/dataprovider/BBM_DataProviderSearch-1.0.qclass.yaml)|performs a search in a record-based data provider and returns the result, includes input/output connectors
|[BBM_DataProviderSearchBase](./generic/dataprovider/BBM_DataProviderSearchBase-1.0.qclass.yaml)|base class for using the data provider search API
74
|[BBM_ExcelReadDataProvider](./generic/dataprovider/BBM_ExcelReadDataProvider-1.0.qclass.yaml)|base building block for processing Excel spreadsheet data as input; implements an output connector
75
76
77
78
79
80
81
82
|[BBM_GenericMapper](./generic/dataprovider/BBM_GenericMapper-1.0.qclass.yaml)|building block for generic data transformation support based on a mapper; includes an input/output connector; use this building block to use a mapper that will **not** run in autonomous mode (for example where input data can be used directly)

### Data Provider / Pipeline Blocks

|Building Block|Description|
|---|---|
|[BBM_GetPipelineData](./generic/dataprovider/BBM_GetPipelineData-1.0.qclass.yaml)|building block for providing a data pipeline with initial processing data from configuration
|[BBM_SimpleFilterPipelineData](./generic/dataprovider/BBM_SimpleFilterPipelineData-1.0.qclass.yaml)|building block for filtering records in a data pipeline
83
84
85
86
87
88
89
90
91
92
93
94
95

### Filesystem Event Building Blocks
|Building Block|Description|
|---|---|
|[BBM_FsEventBase](./service/fsevent/BBM_FsEventBase-1.0.qclass.yaml)|generic class for services for event-driven file actions on the local filesystem

### Generic / Utility Building Blocks
|Building Block|Description|
|---|---|
|[BBM_BindSubworkflow](./generic/BBM_BindSubworkflow-1.0.qclass.yaml)|binds a subworkflow in a subworkflow step; the subworkflow data is provided as input data, includes input/output connectors
|[BBM_BindSubworkflowStatic](./generic/BBM_BindSubworkflowStatic-1.0.qclass.yaml)|binds a subworkflow in a workflow step based on configuration in the building block, includes input/output connectors
|[BBM_CreateOrder](./generic/BBM_CreateOrder-v1.0.qclass.yaml)|creates a new workflow order; the workflow order data is provided as input data, includes input/output connectors
|[BBM_CreateOrderStatic](./generic/BBM_CreateOrderStatic-v1.0.qclass.yaml)|creates a new workflow order based on configuration in the building block, includes input/output connectors
96
|[BBM_DataSerialization](./generic/BBM_DataSerialization-1.0.qclass.yaml)|transforms input data into a serialized string, includes an input/output connector
97
98
99
100
101
102
103
104
105
|[BBM_DeleteFilePath](./generic/BBM_DeleteFilePath-1.0.qclass.yaml)|deletes a file based on configuration, includes an input/output connector
|[BBM_ExecSyncWorkflow](./generic/BBM_ExecSyncWorkflow-1.0.qclass.yaml)|executes a workflow synchronously; the workflow order data is provided as input data, includes input/output connectors
|[BBM_ExecSyncWorkflowStatic](./generic/BBM_ExecSyncWorkflowStatic-v1.0.qclass.yaml)|executes a workflow synchronously based on configuration in the building block, includes input/output connectors
|[BBM_GetArray](./generic/BBM_GetArray-1.0.qclass.yaml)|returns a list of data based on configuration, meant to be used to provide array input for an array step, provides an input/output connector
|[BBM_GetData](./generic/BBM_GetData-1.0.qclass.yaml)|returns arbitrary data based on configuration, provides an input/output connector
|[BBM_InternalIterator](./generic/BBM_InternalIterator-1.0.qclass.yaml)|allows for iteration of internal data; converts a hash with an internal list into a list of hashes having common top-level data with the interior element repeated
|[BBM_JavaConfig](./generic/BBM_JavaConfig-1.0.qclass.yaml)|sets Java configuration based on configuration items
|[BBM_LogMessage](./generic/BBM_LogMessage-1.0.qclass.yaml)|logs a message based on configuration
|[BBM_OutputData](./generic/BBM_OutputData-1.0.qclass.yaml)|writes data to output location according to configuration giving the data to write and the output locations, includes an input/output connector
106
|[BBM_PauseDataPassthru](./generic/BBM_PauseDataPassthru-1.0.qclass.yaml)|executes a configurable and optionally skippable pause or delay, includes an input/output connector
107
|[BBM_QorusServiceMethodCaller](./generic/BBM_QorusServiceMethodCaller-1.0.qclass.yaml)|calls a Qorus service method and returns the result, includes an input/output connector
108
109
|[BBM_RegularExpressions](./generic/BBM_RegularExpressions-v1.0.qclass.yaml)|helper building block used by other building blocks for for regular expression configuration item handling
|[BBM_UpdateOrderDynamicData](./generic/BBM_UpdateOrderDynamicData-1.0.qclass.yaml)|building block that can be used to write data to a workflow order's dynamic data hash, includes an input/output connector
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

### FTP Building Blocks
|Building Block|Description|
|---|---|
|[BBM_FtpPollerBase](./generic/ftp/BBM_FtpPollerBase-v1.0.qclass.yaml)|generic base class for polling for files from an FTP server
|[BBM_FtpPollerCreateOrder](./generic/ftp/BBM_FtpPollerCreateOrder-v1.0.qclass.yaml)|class for polling for files from an FTP server and creating a workflow order from them
|[BBM_FtpPollerCreateOrderJob](./job/ftp/BBM_FtpPollerCreateOrderJob-v1.0.qclass.yaml)|base class for FTP polling jobs that create workflow orders for files received

### Kafka Building Blocks
|Building Block|Description|
|---|---|
|[BBM_KafkaConsumer](./kafka/BBM_KafkaConsumer-1.0.qclass.yaml)|**[Kafka](https://kafka.apache.org/)** message consumer building block, includes input/output connectors
|[BBM_KafkaProducer](./kafka/BBM_KafkaProducer-1.0.qclass.yaml)|**[Kafka](https://kafka.apache.org/)** message producer building block, includes input/output connectors

### HTTP Server Building Blocks
|Building Block|Description|
|---|---|
|[BBM_AwsSnsServiceBase](./service/http/rest/aws/BBM_AwsSnsServiceBase-v1.0.qclass.yaml)|Base IoT AWS SNS service class building block
|[BBM_CorsBase](./service/http/BBM_CorsBase-v1.0.qclass.yaml)|base class for CORS functionality used by other building blocks
|[BBM_HttpAuthenticatorBase](./generic/lib/BBM_HttpAuthenticatorBase-1.0.qclass.yaml)|base class for HTTP authentication used by other building blocks
|[BBM_HttpFileHandler](./service/http/BBM_HttpFileHandler-1.0.qclass.yaml)|generic building block for serving files from the filesystem via HTTP/S
|[BBM_HttpServiceGenericBase](./service/http/BBM_HttpServiceGenericBase-v1.0.qclass.yaml)|base class for HTTP handler services; meant as a base class for higher-level HTTP-based server building blocks
|[BBM_HttpReverseProxyService](./service/http/BBM_HttpReverseProxyService-v1.0.qclass.yaml)|a reverse proxy building block supporting 1:1 forwarding of chunked transfer encoding and protocol handlers such as WebSockets
|[BBM_HttpServiceBase](./service/http/BBM_HttpServiceBase-v1.0.qclass.yaml)|base class for HTTP handler services; can be used directly
|[BBM_QorusUiExtension](./service/http/BBM_QorusUiExtension-1.0.qclass.yaml)|building block for Qorus UI extension server services

136
137
138
139
140
### MQTT Building Blocks
|Building Block|Description|
|---|---|
|[BBM_MqttPahoClient](./mqtt-paho/BBM_MqttPahoClient-v1.0.qclass.yaml)|**[MQTT Paho](https://www.eclipse.org/paho/)** client base building block
|[BBM_MqttPahoPublisher](./mqtt-paho/BBM_MqttPahoPublisher-v1.0.qclass.yaml)|**[MQTT Paho](https://www.eclipse.org/paho/)** client publisher building block with connectors for publishing MQTT messages
141
|[BBM_MqttPahoSubscriber](./mqtt-paho/BBM_MqttPahoSubscriber-v1.0.qclass.yaml)|**[MQTT Paho](https://www.eclipse.org/paho/)** client subscriber building block with an event-based connector for listening to MQTT messages / topics
142

143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
### REST Building Blocks
|Building Block|Description|
|---|---|
|[BBM_RestAction](./generic/rest/BBM_RestAction-v1.0.qclass.yaml)|generic class for making a REST request, includes input/output connectors
|[BBM_RestActionStep](./step/rest/BBM_RestActionStep-v1.0.qclass.yaml)|generic step class for making a REST request, includes input/output connectors
|[BBM_RestActionWithRecovery](./generic/rest/BBM_RestActionWithRecovery-v1.0.qclass.yaml)|generic class for making a REST request, includes recovery logic and config as well as input/output connectors
|[BBM_RestServiceBase](./service/http/rest/BBM_RestServiceBase-v1.0.qclass.yaml)|base class for REST handler services

### SFTP Building Blocks
|Building Block|Description|
|---|---|
|[BBM_SftpPollerBase](./generic/sftp/BBM_SftpPollerBase-v1.0.qclass.yaml)|generic base class for polling for files from an SFTP server
|[BBM_SftpPollerCreateOrder](./generic/sftp/BBM_SftpPollerCreateOrder-v1.0.qclass.yaml)|class for polling for files from an SFTP server and creating a workflow order from them
|[BBM_SftpPollerCreateOrderJob](./job/sftp/BBM_SftpPollerCreateOrderJob-v1.0.qclass.yaml)|base class for SFTP polling jobs that create workflow orders for files received

### Salesforce Building Blocks
|Building Block|Description|
|---|---|
|[BBM_SalesforceStreamBase](./service/salesforce/BBM_SalesforceStreamBase-v1.0.qclass.yaml)|base class for **Salesforce** streaming API support, provides an event source connector
162
|[BBM_SalesforceStreamCreateOrder](./service/salesforce/BBM_SalesforceStreamCreateOrder-v1.0.qclass.yaml)|base class for creating workflow orders based on **Salesforce** events
163
164
165
166
167

### WebSocket Server Building Blocks
|Building Block|Description|
|---|---|
|[BBM_WebSocketServiceBase](service/http/websockets/BBM_WebSocketServiceBase-v1.0.qclass.yaml)|base class for WebSocket handler services
David Nichols's avatar
David Nichols committed
168
|[BBM_WebSocketServiceEventSource](service/http/websockets/BBM_WebSocketServiceEventSource-v1.0.qclass.yaml)|base class for WebSocket handler services with an event connector
169
170
171
172
173
174
175

### WSGi Server Building Blocks
|Building Block|Description|
|---|---|
|[BBM_WsgiHandlerBase](./service/http/wsgi/BBM_WsgiHandlerBase-v1.0.qclass.yaml)|WSGi HTTP server handler base building block; serves any Python-based WSGi server service from a Qorus service, also providing the Qorus API to the WSGi server as well as dynamic Java imports
|[BBM_WsgiServer](./service/http/wsgi/BBM_WsgiServer-v1.0.qclass.yaml)|WSGi HTTP server building block; serves any Python-based WSGi server service from a Qorus service, also providing the Qorus API to the WSGi server as well as dynamic Java imports
|[BBM_WsgiUiExtension](./service/http/wsgi/BBM_WsgiUiExtension-v1.0.qclass.yaml)|Allows for serving a WSGi server as a Qorus UI extension based on configuration