Skip to main content

Databricks Integration

Step-by-step guide to enabling Crossbeam's Databricks Push integration, which writes partner overlap data back to a Delta table in your Databricks workspace.

Written by Joy Rudnick

In this article:

Overview

The Databricks Integration writes the partner overlaps Crossbeam generates back to your Databricks workspace. Crossbeam creates and maintains a single overlaps Delta table in a catalog and schema you specify.

Crossbeam integrates with Databricks in two directions: this article covers the Databricks Integration (Push), which sends partner overlaps back into your Databricks workspace. You can also connect Databricks as a Data Source to sync your account data into Crossbeam. Both are independent and can point to different catalogs and schemas.

🔄 Need to pull data from Databricks into Crossbeam first? See Connect Databricks as a Data Source.

Plan Availability

Free

Connector

Supernode

Enterprise

Databricks Integration

--

--

✅

✅

Prerequisites

Before connecting, confirm you have the following in your Databricks environment:

  • A Databricks workspace with Unity Catalog enabled

  • A SQL Warehouse that Crossbeam can connect to—both values are visible under your SQL Warehouse → Connection Details:

    • Server hostname (e.g. dbc-12345abc-6789.cloud.databricks.com)

    • HTTP path (e.g. /sql/1.0/warehouses/abc123def456)

  • A Service Principal with an OAuth client secret—Crossbeam authenticates as this Service Principal and does not support personal access tokens or interactive OAuth

Creating the Service Principal

In your Databricks account console, navigate to:

  • Settings → Identity and access → Service principals

  • Click Add service principal

  • Open the newly created Service Principal and navigate to Secrets → Generate secret

  • Copy the Client ID (Application ID) and Client Secret—Crossbeam needs both

    • The Client Secret is only shown once. Store it securely before closing the window.

  • Grant the Service Principal CAN USE on your SQL Warehouse:

    • Navigate to SQL Warehouses → your warehouse → Permissions, then add the Service Principal with Can use

Step 1: Create the Destination Catalog and Schema

Create the catalog and schema where Crossbeam will write overlap data. You can use the same catalog and schema as your Sync integration or a different one. The schema must exist before enabling push—Crossbeam will create the overlaps table automatically.

CREATE CATALOG IF NOT EXISTS crossbeam_push;
CREATE SCHEMA  IF NOT EXISTS crossbeam_push.overlaps;

💡Check out these Databricks resources if you need help with creating a catalog and creating a schema.

Step 2: Grant the Service Principal Write Access

Replace crossbeam-sp with the display name or UUID of your Service Principal.

-- Grant catalog access

GRANT USE CATALOG

ON CATALOG crossbeam_push

TO `crossbeam-sp`;

-- Grant schema usage and table creation permissions

GRANT USE SCHEMA, CREATE TABLE

ON SCHEMA crossbeam_push.overlaps

TO `crossbeam-sp`;

-- Grant read/write access to schema objects

GRANT MODIFY, SELECT

ON SCHEMA crossbeam_push.overlaps

TO `crossbeam-sp`;

These privileges allow Crossbeam to:

Privilege

What it allows

CREATE TABLE

Create the overlaps Delta table on the first push

MODIFY

Merge (upsert) and delete overlap records on subsequent pushes

SELECT

Validate the table structure and confirm write permissions

Step 3: Enable Push in Crossbeam

In Crossbeam, navigate to Integrations → Databricks Push → Enable.

Enter the following:

Field

Value

Server hostname

From SQL Warehouse → Connection Details

HTTP path

From SQL Warehouse → Connection Details

Catalog

crossbeam_push

Schema

overlaps

Client ID

Service Principal Application ID

Client Secret

OAuth client secret

Click Enable.

When you enable push, Crossbeam will:

  • Connect using your Service Principal credentials.

  • Run CREATE TABLE IF NOT EXISTS <catalog>.<schema>.overlaps … USING DELTA.

  • Validate that every required column is present.

  • Run a no-op merge to confirm write permissions.

If any of these steps fail, you'll see a specific error in the UI indicating the cause (missing permissions, table not found, etc.).

Destination Table Schema

Crossbeam writes overlap rows to <catalog>.<schema>.overlaps with the following columns:

Column

Type

Description

organization_id

INT

Your Crossbeam organization ID

population_id

INT

Your population ID

population_name

STRING

Your population name

master_id

STRING

Your record ID for the overlapping account

mdm_type

STRING

Record type (account, lead, etc.)

partner_organization_id

INT

Partner's Crossbeam org ID

partner_population_id

INT

Partner's population ID

partner_population_name

STRING

Partner's population name

partner_master_id

STRING

Partner's record ID for the same company

partner_name

STRING

Partner organization name

partner_domain

STRING

Partner organization domain

partner_ae_email

STRING

Partner AE email (if shared)

partner_ae_name

STRING

Partner AE name (if shared)

partner_ae_phone

STRING

Partner AE phone (if shared)

partner_ae_title

STRING

Partner AE title (if shared)

partner_record_name

STRING

Partner's record name

partner_record_website

STRING

Partner's record website

partner_record_type

STRING

Partner's record type

partner_record_country

STRING

Partner's record country

partner_record_industry

STRING

Partner's record industry

partner_record_employees

INT

Partner's record employee count

matched_at

TIMESTAMP

When the overlap was first matched

created_at

TIMESTAMP

When the row was created

updated_at

TIMESTAMP

When the row was last updated

Rows are upserted via MERGE on (population_id, partner_population_id, master_id, partner_master_id). Overlaps that are removed in Crossbeam (e.g., when a population changes) are deleted from this table on the next push.

✍️ Note

To populate partner_ae_name, partner_ae_email, partner_ae_phone, and partner_ae_title, you must include a users table in your Sync integration and add owner_id references on your accounts, deals, and/or leads tables.

Supported Features

Feature

Details

Authentication

OAuth M2M (Service Principal client_id and client_secret)

Destination

A single overlaps Delta table created by Crossbeam in your chosen catalog/schema

Write mode

Incremental MERGE (upsert) on every push

Stale-record cleanup

Yes—overlaps removed in Crossbeam are deleted from your overlaps table

Bookmarking

Per population pair, on updated_at

Table creation

Automatic — CREATE TABLE IF NOT EXISTS … USING DELTA

Trigger

On-demand via Crossbeam—no manual scheduling needed

Limitations

  • Authentication: Service Principal OAuth M2M only. Personal access tokens and interactive OAuth are not supported.

  • Fixed schema: The destination table schema is fixed. Additional columns cannot be added to the overlaps table.

  • Single destination: Push cannot be split across multiple destination tables.

Manage Databricks Integration

In Crossbeam, navigate to Integration→ click on the Settings icon next to your Databricks integration.

From here you can:

  • Enable Data Push: Turn on the push integration to send partner overlap data to your Databricks workspace

  • Customize Data in Push: Control which overlap data and fields are included in each push

  • Manage Partner Populations: Configure which partner Populations are included in your overlap data

  • Partner-Specific Settings: Adjust settings on a per-partner basis to control what data is pushed for each partner relationship

Click Save changes when done.

Related Articles

Related Articles
How to Use the Slack App for Crossbeam
Crossbeam Copilot for Salesforce
Snowflake Integration
How to Set Up the Partner Account CRM Integration
Did this answer your question?