ArkFlow v0.5.0 Release Notes 🎉

We're thrilled to announce the release of ArkFlow v0.5.0! This is a massive update packed with major new features, significant performance enhancements, and a huge number of dependency upgrades to improve stability and security.

The primary focus of this release has been to drastically expand ArkFlow's connectivity and data processing capabilities. We've introduced first-class support for Python scripting, added a wide array of new connectors for databases, cloud storage, and messaging systems, and enhanced the core engine with powerful new features like JOIN operations.

🚀 Highlights

• 🐍 Python Processor Support: You can now embed custom Python scripts directly into your pipelines for flexible and powerful data processing.
• 🔗 Expanded Connectivity: We've added a wealth of new inputs and outputs:
  • Databases: Full support for MySQL and PostgreSQL as output sinks.
  • Messaging & Queues: Added native support for Pulsar (input/output) and enhanced Redis support (cluster mode input, new output module).
  • Cloud & Object Storage: The SQL input now supports reading from major object storage platforms including AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, and HDFS.
  • Industrial IoT: Introduced Modbus input for integrating with industrial devices.
• ⚙️ Core Engine Upgrades:
  • JOIN Operations: You can now perform SQL-style JOIN operations within ArkFlow.
  • Multiple Inputs: Configure and run multiple inputs within a single pipeline for complex data aggregation.
  • Message Ordering: Added support for guaranteeing the order of messages in concurrent processing environments.
• ⚡️ Performance Boosts:
  • Implemented back pressure control in the processor to handle high throughput more gracefully.
  • Optimized Redis operations using pipelines for significant performance gains.
  • Added multi-threaded support for JOIN buffer operations to improve batch processing performance.

✨ What's New

Features

• Processor:
  • Supports Python for custom data transformation logic (#409).
• Inputs:
  • Added support for multiple inputs in one configuration (#389).
  • New file input, separated from the sql input, now supports SQL query filtering (#595, #442).
  • Added modbus input support (#406).
  • Added Redis Cluster mode support (#331).
  • Added object storage support for SQL input: S3, GCS, Azure, HDFS, and HTTP (#382, #384, #386, #387, #410).
• Outputs:
  • Added a native redis output module (#329).
  • Added MySQL and PostgreSQL output support (#297).
• Input/Output:
  • Added pulsar input and output support (#759).
• Core:
  • Added support for JOIN operations (#391).
  • Added support for ordering messages under concurrency (#347).
  • Added a default thread count configuration for pipelines (#348).
• Codec:
  • Added Protobuf support (#459).

Bug Fixes

• fix(json): Fixed a bug where converting JSON to Arrow format resulted in data loss (#325).
• fix(input/http): Fixed a configuration parsing error in the HTTP input (#793).

Refactoring & Internal Improvements

• The Kafka output, codec, and SQL output components have been significantly refactored for better maintainability and performance.
• Removed the Ballista dependency, simplifying the architecture (#532).
• Updated and bumped hundreds of dependencies to their latest versions, ensuring the project is secure and up-to-date.
• Added sccache to GitHub Actions to speed up Rust project builds (#441).

New Contributors

• @duyquang6 made their first contribution in https://github.com/arkflow-rs/arkflow/pull/767

Full Changelog: https://github.com/arkflow-rs/arkflow/compare/v0.3.0...v0.5.0