Breaking Changes

In this Release we decided to improve the Server Landscape, this leads to a couple of major changes in the Setup and Operation of HEDDA.IO.

API Server

We removed the API Server from the Server Landscape. All tasks the API Server handled are now part of the Frontend Server, now referenced as the HEDDA.IO Server. This helps to keep the cost of running and maintaining HEDDA.IO down and reduces the need for duplicating logic.

Note: This requires that the Pyrunner and DotnetRunner now need to be setup with the Frontend URL.

// Prior:
var hedda = Hedda.Create(apiUrl, ...);

// Now:
var hedda = Hedda.Create(frontendUrl, ...);

# Prior:
hedda = Hedda.create(base_url=api_url, ...)

# Now:
hedda = Hedda.create(base_url=frontend_url, ...)

Container App Job / WebRunner

We also switched the design of the Container App Job to be a Full Web Service instead of just a Worker Process. This leads to some other advantages. With now a dedicated Server handling the execution of Validations the HEDDA.IO Server (former Frontend) is now able to utilize it for Preview functionality, as well. Leading to a cost reduction as there is now only the need for one heavy Server handling validation requests instead of two. For it being a Server startup times, of previously around 15sec for external Executions are a thing of the past. Other benefits are that the Executions might also be scaled horizontally if high frequency of parallel executions are necessary.

Edit Version Executions

In the past it was possible to start a validation against a current Edit Version, this is still possible, but with the introduction of named Edit Version the call for this Execution changed slightly.

DotNetRunner

// Prior:
var hedda = Hedda.Create(..., workflowConfig: new(){
  ExecuteAgainstDev= true
})

// Now:
hedda.UseKnowledgeBase(kbName, versionName)

PyRunner

# Prior:
hedda.use_knowledge_base(kb_name, use_current=False)

# Now:
hedda.use_knowledge_base(kb_name, version_name=version_name)

New Features

Git Integration

You can now store and manage your Knowledge Base content directly in a Git repository! This new feature allows you to:

• Version Control: Track changes to your Knowledge Base and easily revert to previous versions.
• Edit Versioning: Multiple users can now work on the same Knowledge Base simultaneously, each on their own version (branch) without impacting each other.
• Multi Environments: You can now configure different Git branches to serve as your Knowledge Bases for different environments (e.g., Development, Staging, Production). This allows for safe testing and deployment of updates before they go live to your users.
• Automation: Integrate your Knowledge Bases into your CI/CD pipelines for automated updates and deployments.

Improvements

Single Row Processing

We integrated the Single Row Processing in the new Web Runner, eliminating the need for the additional Azure function. With the new Architecture we were no longer bound to the stateless nature of Azure functions and could introduce the implementation as a Background Service also eliminating the need for the Mongo Db used for State Caching. As bonus the performance was also increased significantly.

There is no need to change previous Pipelines Calls as the API state basically is the same. But we added two functionalities.

1. Bring Your Own ExecutionId: The ExecutionStart call can now also have the X-Execution-Id Header which will be used to create the Execution.

   Keep in mind that this should no collide with existing Executions

2. If the ExecutionStart Option DisableRowStorage is not set explicitly to true the processed rows will be gathered and are then also available in the Review functionality, later.

Note: The SPR and the Preview functionality are now sharing the same Instance. Keep this in mind when determining the necessary scale of the WebRunner Service

The Dotnet Runner was also extended and can be run directly in Single Row Processing mode enabling it to instantiate once and validate Data as they come in.

var hedda = hedda.Create(frontendUrl, apiKey)
  .UseProject(projectName)
  .UseKnowledgeBase(kbName)
  .UseRun(runName)
  .StartSingle();

var rowResult1 = hedda.ProcessRow([col1, col2, col3]);
var rowResult2 = hedda.ProcessRow([col1, col2, col3]);

var heddaResult = hedda.Finish();

Additionally a ProcessRow Verbose Endpoint /api/processRow/verbose was added which changes the Response slightly to include the Names of the Columns-, Rulebooks- and BusinessRule Results. Instead of the values in an Array the Values are now in an Object with the Keys being the Name of the entity.

Examples:

{
  "RowIndex": 1,
  "Valid": true,
  "Error": null,
  "ColumnResults": {
    "Sales Order ID": {
      "Value": "1ed99882-9d6e-4b5c-b6e2-50b00c987573",
      "OriginalValue": "1ed99882-9d6e-4b5c-b6e2-50b00c987573",
      "PhoneticValue": null,
      "MemberResult": null
    },
    "Customer Name": {
      "Value": "Travis Ryan",
      "OriginalValue": "Travis Ryan",
      "PhoneticValue": null,
      "MemberResult": null
    }
  },
  "RulebookResults": {
    "Product validation": true,
    "Sales Data Check": false
  },
  "BusinessRuleResults": {
    "DataType Validation": true,
    "Member Validation": true
  }
}

Search

We have made significant updates to our global search functionality. The new Full Text Search now includes descriptions, enabling you to find relevant results with greater accuracy.

The Search UI has also been redesigned for improved usability and navigation. Additionally, definitions stored in Git repositories are indexed too.

Key improvements include:

• Enhanced search logic for more accurate results
• Improved user interface for streamlined searching
• Increased search scope with inclusion of definitions from Git repositories

PyRunner

The PyRunner was updated and the pure spark executor was removed the new default executor is the Interop executor which allows for better feature parity with the Dotnet Runner as it is basically utilized under the hood. The Execution performance could be improved vastly with this approach. The external executor is still available and has also received some updates like showing the current execution state and improved error handling.

Some improvements in the interop executor were made in the IPC layer, Utilizing Apache Arrow as transmission format is reducing the communication time by ~20% in the current Spark version 3.5. Coming with Spark 4.0 (currently in Preview) this will get an impressive ~80% boost out of the box. To activate this feature, the Config Option USE_PYARROW_IPC needs to be set to True.

There is only one caveat that PyArrow is not available in every Spark Environment. So far we have identified that it is available in Spark native Environments but not in Spark Connect Environments like Databricks Serverless and Databricks Shared Compute. Databricks Personal Compute and Azure Fabric are supported.

Benchmarked Performance with 4.8M rows (just IPC):

Miscellaneous

• Improved the performance of the Audit queries
• Expanded Domains used in Business Rules to also check Preparations and Actions
• During import of Knowledge Bases, a connection is tested before it is saved or data are loaded
• Improved Environments forms in Excel-Addin and added a button that opens the respective HEDDA Environment
• Alphabetically ordered Member Search Algorithms, Business Rule Actions and Conditions
• Added possibility to order Projects by Name, Owner or Creation Date on the Project Overview Page
• Monitored Run Executions can now be filtered to only show the most recent or last Run Execution
• Monitored Run Executions can now be shown as aggregated statistic tiles
• Add/Edit Connection forms now show the currently selected file
• Added Member Validation and Data Type Validation Filter for Business Rule Filter Options in Preview Page
• PyRunner External Executions are no longer polling for the Result File in the data lake but actually the new WebRunner Service, leading to no more infinite-waiting if execution failed fatally
• It is now possible to create Connections for Data Links while importing a Knowledge Base
• HEDDA.IO + Excel Add-In User Manuals are now available for download on the HELP page under Documentation.
• Improved memory footprint in WebRunner and PreviewRunner

Bugfixes

• Removed Variable Domains from Default Run Mapping
• Fixed a pagination bug in Audit Portal in which the user could set a page lower than 1
• Fixed a bug where the Favorite Button would delete the wrong Favorite
• Fixed a bug where deleting an object would delete all Favorites
• Cell colorization for boolean values in the Excel-Add-In now correctly works independently from the users language settings
• Prevented saving synonyms with empty values in the Member Drawer
• Knowledge Base Import from Provider forms now reset on selecting another provider
• Prevented self-looping Business Rules in Advanced Flow Editor
• Some UI text overflow improvements
• Fixed an error that occurred on save, when replacing the Root Business Rule in Advanced Flow Editor
• Fixed validity icons in Rulebook Flow Display for not executed Rulebooks/Business Rules
• Disallowed selection of a folder, where a file should be selected in Add/Edit Connection forms
• Selecting data from a Range that intersects a table in Excel-Addin, now correctly uses selected data instead of the whole table
• Fixed the Query for items with Data Type Validation Issues in Preview
• Fixed an issue when validating Data Time columns in Single Row Processing