Blockchain Streaming Analytics Brainy Distributed Applications – Blockchainers

Wij are indeed pleased to publish a guest contribution by Kai Wahner about Brainy Distributed Applications.

Kai is Technology Evangelist and Community Director for TIBCO Software. His expertise lies within the fields of Big Gegevens, Advanced Analytics, Machine Learning, Integration, SOA, Microservices, BPM, Cloud, Internet of Things, Blockchain and Programming Languages such spil Java EE, Scala, Groovy, Go or R. He regularly writes about fresh technologies, articles and conference talks on his blog.

Wij are approaching Blockchain Trio.0 thesis days. You ask “3.0”? Gravely? Blockchain is still ter the early adoption phase! Yes, that is true. Nevertheless, while wij are moving forward to real world use cases ter various industries, wij see that you need more than just a blockchain infrastructure. You need to build decentralized applications. Thesis include blockchains, wise contracts plus other applications plus integration and analytics on top of thesis solutions.

Middleware is Key for Success ter Blockchain Projects

Blockchain is the next big thing for middleware! There is no question around this. You need to interconnect other applications, microservices and cloud offerings with a blockchain infrastructure to get real value out of it. Te addition, visual analytics and machine learning have to be leveraged to find insights and patterns te blockchain and non-blockchain gegevens. Eventually, streaming analytics is used to apply thesis insights and patterns to fresh events te a blockchain infrastructure. There is a multiplicity of use cases like fraud detection, compliance issues, optimization of manufacturing or supply chain processes, or any kleuter of screenplays with the Internet of Things (IoT).

Reference Architecture for Blockchain and Middleware

Multitude of Blockchain Platforms including Hyperledger and Ethereum

The blockchain market is growing significantly thesis days. You need to think about various blockchain characteristics for your next blockchain project.

  • Who are the users of the blockchain? Is it public or private? Which vrouwen do you need to work with?
  • Do you want to build your own blockchain infrastructure (based on a framework such spil Hyperledger and one if its implementations like IBM’s Hyperledger Fabric, Iroha, Intel’s Sawtooth Lake) or leverage an existing toneel (like Ethereum)?
  • Do you want to host the infrastructure yourself or leverage a cloud service like IBM’s Bluemix with concentrate on Hyperledger or Microsoft’ Azure with concentrate on Ethereum?
  • What development environment and tooling do you want to use (like Truffle or BlockApps on top of Ethereum)
  • Which characteristics are significant for your script? What about speed, security, overeenstemming algorithms, integration of non-blockchain services, and other significant aspects? Maybe an add-on on top of a blockchain is needed, like the Raiden Network to leverage off-chain state networks to extend Ethereum with some nice properties like scalability or high spectacle for asset transfers?
  • Or do you want to concentrate on a industry-specific blockchain solution like R3 Corda or Ripple for financial services?
  • What middleware do you need? Do you need Application Integration or API Management to interconnect everything? Visual Analytics to find insights and patterns te historical blockchain gegevens? Streaming Analytics to apply rules to activity ter (near) real time for fresh blockchain events?

The following shows how to leverage Streaming Analytics together with blockchain events. This example uses TIBCO StreamBase ter conjunction with the public Ethereum test network. Note that similar scripts can be build with any other blockchain infrastructure. A follow-up postbode about how to leverage middleware with Hyperledger will come soon, too.

Streaming Analytics for Correlation of Blockchain and Non-Blockchain Events

The screenplay uses a Brainy Contract to define a Coin system. You can mine coins and transfer them to other users (i.e. blockchain addresses). This example is similar to Bitcoin concepts to showcase how to leverage streaming analytics with any custom-built blockchain application and clever contracts. The objective is not to display the power of brainy contracts (other articles are available for this). The programming language used to develop this Brainy Contract is Solidity, more or less the den facto standard to write clever contracts for Ethereum.

Here is the Clever Contract built and deployed with Browser Solidity:

Wise Contract ‘Coin’ developed and deployed with Browser Solidity

MetaMask, a bridge to run Ethereum dApps ter your Chrome browser, is running ter the background to connect to the Ethereum network and commit the transactions developed with Browser Solidity. You could also use Streaming Analytics to deploy clever contracts, of course. However, ter this example TIBCO StreamBase wasgoed only used for the following two parts:

  • Receive fresh events from the blockchain network: You can filterzakje, aggregate, analyseren or convert any events like pending transactions, logs or blockchain blocks – and also combine this information with non-blockchain events, of course. For example, you could build a streaming analytics process to analyze just the logs relevant for your specific transaction IDs to spot issues and act proactively, let’s say if a pending transaction takes too long or fails.
  • Execute transactions on the blockchain network via brainy contracts: You can mine fresh coins, send coins to other blockchain addresses and also check the balance of an address. Anything what the clever contract permits can be included into the streaming analytics process.

The streaming analytics process monitors all Ethereum events continuously. This is not spil trivial spil you might know it from classical messaging systems. You cannot just listen to a topic or queue, but you have to pull information out of the blockchain. Depending on the use case, you have to implement some solution which solves your problem but also does not consume too many resources. This is always a trade-off, which has to be thought through when building your streaming analytics process. This also very depends on the blockchain infrastructure you use and its feature set.

Please note that security considerations are not part of this example. Ter the real world, you would integrate encryption and other security requirements into the streaming process, of course. Te this demo, wij use “hardcoded” private keys for sending transactions. A no-go te a real world project.

Let’s now take a look at an implementation of this process.

TIBCO StreamBase + Ethereum Blockchain

Here is the demo setup:

Architecture with TIBCO StreamBase + Ethereum Blockchain

The Ethereum test network is a distributed peer to peer ledger. It runs on various Ethereum clients. Wij used one of the most mature ones on our local laptop: The geth client implement ter Golang. This is synced and also part of the Ethereum test network.

TIBCO StreamBase is used to build the streaming analytics process:

TIBCO StreamBase Connectors for Ethereum

The web3j Java API is used to connect TIBCO StreamBase with the Ethereum network through our local geth client. You just need to write the connector once and can reuse it ter all your streaming processes via haul&,druppel and configuration. Thesis behave “just” like any other connector (such spil messaging via MQTT or Apache Kafka) and components to build streaming logic (like filterzakje, aggregate or convert).

Building this process wasgoed actually a pretty effortless task with TIBCO StreamBase. Te the same way, you can build much more sophisticated blockchain processes te your real world project. Let’s also think about some other next steps.

Next Steps: Application Integration, API Management, Machine Learning, and more

A real world blockchain projects needs streaming analytics to correlate blockchain and non-blockchain events to fight fraud or compliance issues, to improve efficiency ter manufacturing or supply chain processes, to combine Internet of Things with blockchain infrastructures, and for many other use cases.

However, Streaming Analytics is just one lump of the puzzle. Here are some more thoughts about why you might combine blockchain with middlware and analytics:

  • Live Visualization for Real Time Monitoring and Proactive Deeds
  • Cross-Integration with Ethereum and Hyperledger Blockchains
  • Gegevens Discovery for Historical Analysis to Find Insights and Patterns
  • Machine Learning to Build of Analytic Models
  • Application Integration with other Applications (Legacy, Cloud Services, …)
  • API Management to expose blockchain services and treat caching / throttling challenges
  • Native Hardware Integration with Internet of Things Devices

I will do more posts about thesis ideas and display more live demos ter the next weeks and months. Ter the meantime, very first customer projects also kicked off, already. Blockchain and middleware have a excellent and interesting future…

Blockchain, Ethereum, Hyperledger, Middleware, Integration, Streaming Analytics, TIBCO, StreamBase, Live Datamart, Brainy Contracts, Cloud, web3j

Related movie: Bitcoin Gold Mining: Difficulty Adjustment Algorithm


Leave your comment