User-defined operators in Lakeflow Designer

Lakeflow Designer lets you create user-defined operators that appear directly in the canvas alongside built-in operators. Use them to extend Lakeflow Designer with your own business logic, calculations, or integrations.

User-defined operators come in two scopes:

• Local: The operator belongs to one visual data prep file. Describe the transformation, and Genie Code generates the operator. See Create a local user-defined operator with Genie Code.
• Shared: The operator is registered in the workspace, so you and other users can add it to any visual data prep file. Its logic is a Python function or a Unity Catalog function, described in YAML. Write it yourself, or ask Genie Code to create it. See How user-defined operators work.

Create a local user-defined operator with Genie Code

When Genie Code is available in your workspace, you can create a local user-defined operator from the Lakeflow Designer canvas:

1. In the Operators pane, select Create user-defined operator.

2. Enter a description and, optionally, a name. Select Generate preview.

   Tip

   For example, enter the following description:

   Count the business days between two date columns, excluding weekends. Let me choose the start and end date columns and name the output column.
   

3. Review the generated configuration and Python function, then select Approve and create.

Designer stores the local operator's definition in the current file and adds it to the Operators pane. Use the operator's actions menu to edit or delete the definition. You must remove every instance of the operator from the canvas before you can delete its definition.

Choose a shared operator type

Shared user-defined operators have three types:

• python-run-function: A standalone YAML file with inline Python stored in the workspace. Best for DataFrame-level transformations and external integrations. Permissions are managed at the workspace file level.
• uc-udf: Wraps a Unity Catalog scalar function. Best for column-level transformations. Access is governed by Unity Catalog permissions.
• uc-udtf: Wraps a Unity Catalog table-valued function. Best for table-level transforms like ML clustering and aggregation. Access is governed by Unity Catalog permissions.

How user-defined operators work

A shared user-defined operator consists of:

• Operator logic: The code that runs when the operator executes. This is an inline Python run() function for python-run-function operators, or a Unity Catalog function for uc-udf and uc-udtf operators. See Write logic for Lakeflow Designer user-defined operators.
• YAML configuration: Tells Lakeflow Designer how to present the operator in the UI, including the operator's name, description, input parameters, UI widgets, and ports. All operator types use the user-defined-operator-v0.1.0 schema. For a python-run-function operator, the YAML is a standalone workspace file that also holds the inline code. For a Unity Catalog function, the YAML is embedded in the function body. See User-defined operator YAML reference.
• Registration file: An entry in .user_defined_operators.yaml that lets Lakeflow Designer discover the operator. See Register and deploy your operator to Lakeflow Designer.

Build a shared operator

To build a shared user-defined operator from start to finish:

1. Choose an operator type. See Choose a shared operator type.
2. Write and test the operator logic. See Write logic for Lakeflow Designer user-defined operators.
3. Describe the operator in YAML: its name, configuration properties, UI widgets, and ports. See User-defined operator YAML reference.
4. Register the operator in .user_defined_operators.yaml. See Register and deploy your operator to Lakeflow Designer.
5. Grant permissions so other users can discover and run the operator. See Permissions.
6. Use the operator. It appears in the Operators pane in Lakeflow Designer, alongside the built-in operators.

Each tutorial walks through these steps for one complete operator.

Register and deploy your operator to Lakeflow Designer

For your operator to appear in Lakeflow Designer, register it in a .user_defined_operators.yaml file:

• Workspace level: Place the file in the root of your workspace to make the operator visible to all users.
• User level: Place the file in your user home folder (/Workspace/Users/<user-name>/.user_defined_operators.yaml) to make operators visible only to you.

The operators: section supports file paths, Unity Catalog function references, and glob patterns. You can mix entry types:

operators:
  # File path (python-run-function operators)
  - /Workspace/Users/me/udos/my_operator.yaml
  # Glob pattern (registers all matching files)
  - /Workspace/Users/me/udos/transforms/*.yaml
  # UC function reference (uc-udf and uc-udtf operators)
  - catalog: my_catalog
    schema: my_schema
    functionName: my_function

Update or remove an operator

When you change an operator's code, refresh your user-defined operators to load the change. In the Operators tab of the menu, click .

• If the operator keeps the same version, refreshing loads the updated code.
• If the operator has a new version, the operator on the canvas prompts you to upgrade to it (or keep the current version) after you refresh.

To remove an operator from Lakeflow Designer, delete its entry from .user_defined_operators.yaml. For uc-udf and uc-udtf operators, you can also drop the underlying Unity Catalog function with DROP FUNCTION if you no longer need it.

Permissions

Troubleshooting

Additional resources

Explore the following tutorials:

To write the operator logic, see Write logic for Lakeflow Designer user-defined operators. For a complete reference to the YAML schema, see User-defined operator YAML reference.