Tutorial: Gmail email sender operator

In this tutorial, you create a python-run-function operator for Lakeflow Designer that sends the contents of a DataFrame as a CSV attachment through Gmail. Use this example to learn how to build YAML-based operators that perform side effects, such as sending notifications or writing to external systems. To learn more, see User-defined operators in Lakeflow Designer.

Overview

The operator reads the Gmail App Password from a Azure Databricks secret scope. It passes the input DataFrame through unchanged to its output port, so you can chain additional operators downstream. During a workflow preview, it sends no email.

Requirements

• A Azure Databricks workspace with access to create secret scopes.
• A Gmail account with a Google App Password (required when multifactor authentication (MFA) is enabled).
• The Databricks CLI installed on your local development machine.

Step 1: Set up secrets

Store your Gmail credentials in a Azure Databricks secret scope so the operator can retrieve them at runtime.

1. Create a secret scope using the Azure Databricks CLI:

   databricks secrets create-scope my_email_scope
   

2. Store your Gmail App Password in the scope:

   databricks secrets put-secret my_email_scope gmail_app_password
   

   You're prompted to enter the secret value. Paste your Gmail App Password and save.

Step 2: Write the run() function

The python-run-function operator type requires a run() function with this signature:

def run(config: Dict[str, Any], inputs: Dict[str, Any], spark) -> Dict[str, Any]:

• config: Configuration values provided by the user in the Lakeflow Designer UI.
• inputs: Input DataFrames keyed by port name.
• spark: The active Spark session.

The function must return a dictionary of output DataFrames keyed by output port name.

Define and test the function in a notebook cell. When is_preview is set, the function skips its side effects and returns the input unchanged:

from typing import Dict, Any

def run(config: Dict[str, Any], inputs: Dict[str, Any], spark) -> Dict[str, Any]:
    input_df = inputs["data"]

    if config.get("is_preview", False):
        return {"data": input_df}

    import smtplib
    import os
    from email.mime.multipart import MIMEMultipart
    from email.mime.text import MIMEText
    from email.mime.base import MIMEBase
    from email import encoders

    sender_email = config.get("sender_email", "")
    secret_scope = config.get("secret_scope", "")
    secret_key = config.get("secret_key", "")
    recipients_raw = config.get("recipients", "")
    subject = config.get("subject", "")
    body = config.get("body", "")

    if not sender_email:
        raise ValueError("Sender Email is required.")
    if not secret_scope or not secret_key:
        raise ValueError("Secret Scope and Secret Key are required.")
    if not recipients_raw:
        raise ValueError("At least one recipient is required.")

    recipients = [r.strip() for r in recipients_raw.split(",") if r.strip()]
    if not recipients:
        raise ValueError("At least one valid recipient email is required.")

    from pyspark.dbutils import DBUtils
    dbutils = DBUtils(spark)
    sender_password = dbutils.secrets.get(scope=secret_scope, key=secret_key)

    pdf = input_df.toPandas()
    file_path = "/tmp/designer_email_attachment.csv"
    pdf.to_csv(file_path, index=False)

    for recipient in recipients:
        msg = MIMEMultipart()
        msg["From"] = sender_email
        msg["To"] = recipient
        msg["Subject"] = subject
        msg.attach(MIMEText(body, "plain"))

        with open(file_path, "rb") as attachment:
            part = MIMEBase("application", "octet-stream")
            part.set_payload(attachment.read())
            encoders.encode_base64(part)
            part.add_header(
                "Content-Disposition",
                f"attachment; filename={os.path.basename(file_path)}",
            )
            msg.attach(part)

        with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
            server.login(sender_email, sender_password)
            server.send_message(msg)

    if os.path.exists(file_path):
        os.remove(file_path)

    return {"data": input_df}

Step 3: Test the function

Test the function with a sample DataFrame in preview mode, which returns the input DataFrame unchanged and sends no email. Replace <your-email@gmail.com> with your Gmail address.

test_df = spark.createDataFrame(
    [("Alice", 100), ("Bob", 200)],
    ["name", "amount"]
)

result = run(
    config={
        "is_preview": True,
        "sender_email": "<your-email@gmail.com>",
        "secret_scope": "my_email_scope",
        "secret_key": "gmail_app_password",
        "recipients": "alice@example.com",
        "subject": "Test",
        "body": "Test body"
    },
    inputs={"data": test_df},
    spark=spark
)

result["data"].show()

Step 4: Build the YAML definition

Create a file called gmail_email_sender.yaml with the following content:

schema: user-defined-operator-v0.1.0
id: gmail_email_sender
type: python-run-function
version: '1.0.0'
name: Gmail Email Sender
description: Sends the input DataFrame as a CSV attachment via Gmail SMTP to one or more recipients.

config:
  type: object
  properties:
    is_preview:
      type: boolean
      format: is_preview
      default: false
    sender_email:
      type: string
      title: Sender Email
      default: ''
      examples:
        - 'you@gmail.com'
      x-ui:
        widget: input
    secret_scope:
      type: string
      title: Secret Scope
      default: ''
      examples:
        - 'my_email_scope'
      x-ui:
        widget: input
    secret_key:
      type: string
      title: Secret Key
      default: ''
      examples:
        - 'gmail_app_password'
      x-ui:
        widget: input
    recipients:
      type: string
      title: Recipients
      default: ''
      examples:
        - 'alice@example.com, bob@example.com'
      x-ui:
        widget: textarea
        rows: 2
    subject:
      type: string
      title: Subject
      default: ''
      examples:
        - 'Designer Output Data'
      x-ui:
        widget: input
    body:
      type: string
      title: Email Body
      default: "Hello,\n\nAttached is the latest data.\n\nBest,\nDatabricks Workflow"
      x-ui:
        widget: textarea
        rows: 6
  required:
    - sender_email
    - secret_scope
    - secret_key
    - recipients
    - subject
  additionalProperties: false

ports:
  input:
    - name: data
      title: Input Data
      mime: application/vnd.databricks.dataframe
  output:
    - name: data
      title: Output Data
      mime: application/vnd.databricks.dataframe

run_function:
  type: inline
  code: |
    from typing import Dict, Any

    def run(config: Dict[str, Any], inputs: Dict[str, Any], spark) -> Dict[str, Any]:
        input_df = inputs["data"]

        if config.get("is_preview", False):
            return {"data": input_df}

        import smtplib
        import os
        from email.mime.multipart import MIMEMultipart
        from email.mime.text import MIMEText
        from email.mime.base import MIMEBase
        from email import encoders

        sender_email = config.get("sender_email", "")
        secret_scope = config.get("secret_scope", "")
        secret_key = config.get("secret_key", "")
        recipients_raw = config.get("recipients", "")
        subject = config.get("subject", "")
        body = config.get("body", "")

        if not sender_email:
            raise ValueError("Sender Email is required.")
        if not secret_scope or not secret_key:
            raise ValueError("Secret Scope and Secret Key are required.")
        if not recipients_raw:
            raise ValueError("At least one recipient is required.")

        recipients = [r.strip() for r in recipients_raw.split(",") if r.strip()]
        if not recipients:
            raise ValueError("At least one valid recipient email is required.")

        from pyspark.dbutils import DBUtils
        dbutils = DBUtils(spark)
        sender_password = dbutils.secrets.get(scope=secret_scope, key=secret_key)

        pdf = input_df.toPandas()
        file_path = "/tmp/designer_email_attachment.csv"
        pdf.to_csv(file_path, index=False)

        for recipient in recipients:
            msg = MIMEMultipart()
            msg["From"] = sender_email
            msg["To"] = recipient
            msg["Subject"] = subject
            msg.attach(MIMEText(body, "plain"))

            with open(file_path, "rb") as attachment:
                part = MIMEBase("application", "octet-stream")
                part.set_payload(attachment.read())
                encoders.encode_base64(part)
                part.add_header(
                    "Content-Disposition",
                    f"attachment; filename={os.path.basename(file_path)}",
                )
                msg.attach(part)

            with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
                server.login(sender_email, sender_password)
                server.send_message(msg)

        if os.path.exists(file_path):
            os.remove(file_path)

        return {"data": input_df}

Step 5: Save and register the operator

1. Save the YAML file to your Azure Databricks workspace. For example:

   /Workspace/Users/<user-name>/gmail_email_sender.yaml
   

2. Add the operator to your .user_defined_operators.yaml file:

   operators:
     - /Workspace/Users/<user-name>/gmail_email_sender.yaml
   

For more information about registration options, see Register and deploy your operator to Lakeflow Designer.

Step 6: Set up permissions

Users who run a workflow containing this operator need READ access to the secret scope, or they can provide their own secret scope and key values in the operator configuration. Users also need read access to the YAML file in the workspace.

To grant access to the secret scope:

databricks secrets put-acl my_email_scope <user-or-group> READ

Step 7: Use the operator in Lakeflow Designer

1. In a visual data prep, open the Operators menu in the left panel and find the Gmail Email Sender operator.
2. Drag the operator onto the canvas, and connect an input to it.
3. Double-click the operator to open its configuration pane, then set its fields.
4. Select the operator to preview its results in the output pane.

The operator has an input port for your data source, and configuration fields for sender email, secret scope, secret key, recipients, subject, and body.

Additional resources

• User-defined operators in Lakeflow Designer
• Write logic for Lakeflow Designer user-defined operators
• User-defined operator YAML reference