Skip to main content

​
Introduction

Databricks is a platform that allows organizations to store, analyze, and process large volumes of structured and semi-structured data in a highly scalable and efficient manner. With its unique architecture, organizations can consolidate their data, perform quick analytics, and gain valuable data-driven insights accessible to all users.

​
MoEngage <> Databricks

With this integration, you can export data from campaign interaction events to your Databricks tables.

​
Delivery Methods

MoEngage delivers event data to Databricks in two ways. Both methods produce the same table schema and refresh hourly. OpenShare (Zero-Copy Sharing) is the recommended method. It needs no credentials and no IP allowlisting, and your workspace reads the data in place instead of receiving copies. Use a direct export when your workspace does not have Unity Catalog enabled, or when you need MoEngage to write events into a table you own.
The two methods use separate storage layers. Records already delivered through a direct export do not appear in an OpenShare catalog, and vice versa. To move between methods, contact your MoEngage Customer Success Manager (CSM).

​
Integration

​
Prerequisites

The prerequisites depend on the delivery method you choose. Click the tab for your method:
  • Ensure Unity Catalog is enabled on your Databricks workspace.
  • Retrieve your sharing identifier, formatted as <cloud>:<region>:<metastore-id>. Step 1 explains where to find it.
  • Know your primary storage region, which you provide along with the sharing identifier.

​
Set Up Your Export

  • For the full list of events and attributes, refer to the Data Export Glossary.
  • The event_time of the exported events is always in UTC.
  • MoEngage does not support exporting historical data. Data flows from the point you enable the export.
The setup steps differ by delivery method. Click the tab for the method you are using:
With OpenShare, your workspace queries MoEngage-hosted data directly through the Delta Sharing protocol instead of receiving file-based exports. You do not exchange credentials or write tokens, and your queries always read the current data.The following diagram shows how data moves between MoEngage and your workspace:Architecture diagram showing MoEngage provisioning an isolated catalog, share, and recipient, connected to your Databricks workspace over a secure Delta Share that you mount as a read-only catalog and query liveStep 1: Request EnablementTo request OpenShare, perform the following steps:
  1. In your Databricks workspace sidebar, click Catalog.
  2. Click Share, and then click OpenSharing. Databricks Catalog page with the Share menu open, showing the OpenSharing option
  3. On the OpenSharing page, click Sharing ID to copy your sharing identifier. Databricks OpenSharing page with the Sharing ID button highlighted
  4. Send the sharing identifier and your primary storage region to your MoEngage CSM.
Step 2: Accept the ShareAfter MoEngage provisions the share, which usually takes up to seven business days, your MoEngage representative sends you the name of your data share. To accept the share, perform the following steps:
  1. In your Databricks workspace, click Catalog > OpenSharing, and then click the Shared with me tab. Databricks OpenSharing page with the Shared with me tab highlighted, listing the available providers
  2. Click the MoEngage provider, identified by the name your CSM sent you.
  3. Next to the share, click Mount to catalog. Databricks provider page showing the MoEngage share with the Mount to catalog button highlighted
  4. When prompted for a catalog name, enter a name of your choice, for example, moengage_shared_data.
  5. Confirm the catalog to create it.
Step 3: Configure Your ExportTo configure the export in MoEngage, perform the following steps:
  1. On the left navigation menu in the MoEngage dashboard, click App marketplace, search for Databricks, click the Databricks tile, and then click the Integrate tab.
  2. Skip the connection setup. OpenShare provisions the connection for you, so MoEngage does not prompt you for service principal credentials or write access.
  3. Create the export by following Add a New Databricks Export on the MoEngage App Marketplace, below.
  4. In Export into schema, select the shared catalog’s schema. The dropdown shows the same value your CSM shared with you.
Step 4: Query Your DataThe first rows can take up to 30 minutes to appear in your shared catalog. To verify the data, perform the following steps:
  1. Run the following query, replacing the placeholders with your catalog, schema, and table names:
  2. Use Catalog Explorer to browse the new catalog and view the available tables. Databricks Catalog Explorer showing the mounted moengage_shared_data catalog under Shares received, with its schemas and tables
  3. Open the Sample Data view to preview records and attribute definitions.
If you want MoEngage to walk you through the schema, request a technical schema review from your CSM.Step 5: Manage Access Within Your OrganizationBecause the shared catalog sits in your own infrastructure, you control access through Unity Catalog. Assign the USE CATALOG and SELECT privileges to the users or groups that need the data.MoEngage has no write access to the catalog, and either party can end the connection at any time. Tell your CSM about any changes to your workspace setup or sharing identifier, because these changes can interrupt delivery.

​
Add a New Databricks Export on the MoEngage App Marketplace

Both delivery methods use this step. With OpenShare the connection already exists, so you only choose the schema, events, and attributes. To add a new Databricks export, perform the following steps:
  1. On the left navigation menu in the MoEngage dashboard, click App marketplace.
  2. On the App marketplace page, search for Databricks. MoEngage App Marketplace search results showing the Databricks tile
  3. Click the Databricks tile.
  4. On the Databricks page, click the Integrate tab, and then click + Add Export.
  5. Enter the following details: MoEngage Databricks export configuration form showing the export name, connection, schema, and event selection fields
  6. Click Save export.
After you save the export, data refreshes every hour. For a direct export, MoEngage appends events to a table in your schema. For OpenShare, MoEngage refreshes the shared datasets that your workspace queries.

​
Export Process and Frequency

Both delivery methods refresh hourly, and neither requires you to run manual sync jobs. For a direct export, MoEngage creates a destination table in your schema after you save the export, and then appends new event data at hourly intervals:
  • Initial action: MoEngage creates the table when the first export is triggered.
  • Recurring action: MoEngage appends data to the same table every hour.
For OpenShare, MoEngage refreshes the shared datasets every hour. Your queries read the current state of the data, so there is no table for MoEngage to write into on your side.

​
Sample Table Structure

MoEngage creates a single table in your schema using the following format:
moe_events_<table_creation_timestamp>. Note: This table is created per connection. All hourly exports and different event types configured under this connection are consolidated and appended to this specific table. The overall structure of the table remains fixed for all event types. Any new user property or event attribute that is exported is included in the user_attributes or event_attributes column, respectively. All events are consolidated in a single table containing necessary information, such as event names and event times, for easy analysis: The schema is the same for both delivery methods. Custom column naming is not supported.

​
Frequently Asked Questions

It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.
Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.
As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports. This applies to both direct exports and OpenShare.
If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.
No, Data Exports is primarily built to export your Events in near real-time and user attributes in MoEngage are updated asynchronously so it is currently not possible to guarantee the latest values of user attributes in the exports.
Yes. Contact your MoEngage CSM to change delivery methods. The two methods use the same schema but separate storage layers, so records already delivered through your direct export do not appear in the OpenShare catalog.
No. The schema is standardized and custom naming is not supported. You choose which events and attributes to export during configuration, as described in Step 3 of the OpenShare setup.
Your MoEngage CSM is the primary contact for maintaining the connection, resolving technical questions, and adjusting your data requirements.