Skip to main content
Version: v2.10.0

Notebook Advanced Features

Overview​

Yeedu notebooks include advanced features for performance optimization, configuration management, and cluster integration.

Session Management​

Auto-Stop on Cluster Error​

Notebooks automatically stop when the cluster enters an error state:

  • Prevents data loss from queries
  • Clears resources immediately
  • Preserves notebook state for recovery
  • Notification displayed to user

Manual Session Control​

  1. Open notebook session menu
  2. View active cluster
  3. Options available:
    • Detach From Cluster: Keep notebook, stop using cluster
    • Restart Session: Clear variables, start fresh
    • Stop Session: Save notebook, close session

Turbo Mode​

Enable/disable Turbo optimization for faster processing:

Enable Turbo​

  1. From cluster dropdown in notebook editor
  2. Click Turbo Enable/Disable
  3. Toggle Turbo Mode on
  4. Optimizations applied immediately

Turbo Features​

  • Optimized execution time
  • Better resource utilization
  • Recommended for large queries
  • May use additional memory

Max Concurrency​

Control parallel task execution:

Configure Max Concurrency​

  1. In notebook create/edit form
  2. Find Max Concurrency field
  3. Set number (e.g., 4, 8, 16)
  4. Controls parallel task count
  5. Save configuration

Concurrency Guidance​

  • Low (1-2): Sequential, minimal resources
  • Medium (4-8): Balanced parallelism
  • High (16+): Maximum parallelism (more resources)

Impact​

  • Higher concurrency = faster execution
  • Higher concurrency = more resources needed
  • Too high may cause resource contention
  • Fine-tune based on cluster size

Create Cluster from Notebook​

Create a new cluster directly without leaving notebook:

Create Cluster Button​

  1. In notebook toolbar
  2. Click Cluster Selector dropdown
  3. Select Create New Cluster
  4. Quick cluster creation dialog opens

Configuration​

  1. Cluster Name: Identifier for new cluster
  2. Cluster Size: Small, medium, large
  3. Cloud Environment: Select where to create
  4. Click Create to provision

Automatic Attachment​

  1. New cluster created in background
  2. Notebook attached automatically upon completion
  3. Continue working once ready
  4. Monitor creation in cluster status

dbutils.widgets Panel​

Render dbutils widgets in the Widgets Panel:

What are dbutils.widgets?​

Widgets allow interactive parameters in notebooks:

dbutils.widgets.text("param_name", "default_value", "Label")

Widget Rendering​

  1. After cell execution with widgets
  2. Widgets Panel appears (right sidebar)
  3. Shows all defined widgets:
    • Text input
    • Dropdown select
    • Date picker
    • Range slider

Using Widgets​

  1. Set widget values in Widgets Panel
  2. Reference in code: dbutils.widgets.get("param_name")
  3. Run cells to use updated values
  4. Great for parameterized notebooks

Example Workflow​

# Define widgets
dbutils.widgets.text("customer_id", "1", "Customer ID")

# Use widget value
customer_id = dbutils.widgets.get("customer_id")

# Query using parameter
df = spark.sql(f"""
  SELECT * FROM customers 
  WHERE id = {customer_id}
""")

Download Notebook Run Data Snapshots​

Export notebook execution data for analysis:

Create Snapshot​

  1. Execute notebook cells
  2. Click File menu
  3. Select Download Run Snapshot
  4. Choose data to include:
    • Cell outputs
    • Variable states
    • Execution time
    • Resource metrics

Snapshot Contents​

The snapshot includes:

  • Cell code and execution order
  • Cell outputs (text, tables, plots)
  • Variable values (snapshots at time of export)
  • Execution metadata
  • Notebook structure

Use Cases​

  • Share results with team
  • Reproducibility documentation
  • Historical analysis
  • Backup execution state
  • Offline review

File Format​

Snapshots exported as:

  • JSON: Structured data format
  • HTML: Interactive viewer
  • ZIP: Complete package with artifacts

Advanced Notebook Options​

For Scala Notebooks​

Scala now supports advanced options:

  • Packages: Add Maven dependencies
  • Jars: Include JAR files
  • Files: Reference configuration files
  • Conf: Spark configuration parameters
  • Metastore Configuration: Catalog settings

Configuration Example​

// In Advanced Options:
// Packages: org.apache.commons:commons-math3:3.6.1
// Conf: spark.executor.memory=4g

import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics

For Python and SQL​

Similar advanced options available:

  • Packages: pip packages
  • Conf: Spark parameters
  • Environment: Custom environment variables

Notebook Lock Status​

Notebook Locking​

Notebooks can be locked by:

  • Author for protection
  • Admin for governance
  • Temporary during important work

Lock Indicators​

  • 🔒 Locked: Cannot edit
  • 🔓 Unlocked: Can edit
  • ⊘ Read-Only: View only

Action Availability​

  • Locked notebooks show Request Edit button
  • Request sent to notebook owner
  • Owner can unlock for editing
  • Actions disabled based on permissions

Variable Inspector​

Monitor variables and their state:

Open Inspector​

  1. Click Variables panel
  2. See all current variables:
    • Variable name
    • Type (int, string, DataFrame, etc.)
    • Current value (if simple)
    • Memory size (for large objects)

Using Inspector​

  • Verify variable values
  • Check data types
  • Debug variable issues
  • Monitor DataFrame schemas
  • Review cached datasets

Collaborative Editing​

Multiple users can work on same notebook:

Sharing a Notebook​

  1. Click Share button
  2. Add users or groups
  3. Set permissions:
    • View: Read-only
    • Edit: Can modify
    • Admin: Can share and change settings

Real-Time Collaboration​

  • Changes visible to all users
  • Conflict resolution for simultaneous edits
  • Comments for discussing code
  • Execution tracked per user

Performance Tips​

✓ For Better Performance:

  • Clear outputs regularly
  • Use appropriate data formats (Parquet > CSV)
  • Partition large datasets
  • Monitor cell execution time
  • Use Turbo mode for heavy queries
  • Cache frequently accessed DataFrames

✗ Avoid:

  • Very large DataFrames in memory
  • Unoptimized Spark queries
  • Too many small partitions
  • Excessive widget refreshes

See Also​

  • [Create Notebook](./Create notebook.md)
  • [Editor Pane](./Editor Pane.md)
  • [Notebook Types](./Notebook types.md)
  • Auto-Save