Skip to main content
Version: v2.10.0

Spark Logs in Notebook

Overview​

View Spark logs directly within the notebook editor. Monitor Spark driver and executor logs without leaving your notebook environment.

Accessing Spark Logs​

From Notebook Editor​

  1. Open a notebook with running Spark code
  2. Click Spark Logs panel (right sidebar)
  3. View live log output
  4. Filter by log level

During Job Execution​

  1. Cell executes Spark code
  2. Spark Logs tab becomes active
  3. Real-time log output displays
  4. Scroll for older log entries
  5. Auto-scroll follows latest logs

Log Types​

Driver Logs​

Logs from the Spark driver (main process):

  • Job initialization
  • Task scheduling
  • Result collection
  • Error handling
  • Stage progression

Executor Logs​

Logs from Spark executors (worker processes):

  • Task execution details
  • Data processing steps
  • Cache operations
  • Spill to disk events
  • GC activity

Filtering Logs​

Filter by Log Level​

LevelUsage
DEBUGDetailed debugging info
INFOGeneral informational messages
WARNWarning messages (non-critical)
ERRORError messages (critical)
FATALFatal errors (crash)

Apply Level Filter​

  1. Open Spark Logs panel
  2. Click Filter dropdown
  3. Select log level:
    • Show only WARN and ERROR
    • Show only ERROR and FATAL
    • Show all levels (default)
  4. Logs filtered in real-time

Searching Logs​

Search for Keywords​

  1. In Spark Logs panel
  2. Click Search field
  3. Enter keyword or phrase
  4. Matching lines highlighted
  5. Navigate results with arrow buttons

Common Search Terms​

  • "Shuffle": Shuffle operations
  • "Task": Task execution
  • "Stage": Spark stages
  • "Error": Error messages
  • "Exception": Exceptions

Interpreting Common Messages​

Job Initialization​

INFO SparkContext: Starting job:
INFO DAGScheduler: Registering RDD
INFO BlockManager: Block broadcast_0 stored

These messages indicate Spark job is starting.

Task Execution​

INFO Executor: Running task 0.0 in stage 0.0
INFO Task: Task 0.0 finished
INFO Task: Task 0.1 finished

Shows progress of task execution across executors.

Shuffle Operations​

INFO BlockManager: Created local directory
INFO SortShuffleManager: Creating shuffle writer for shuffle 0
INFO BlockStoreShuffleReader: Reading shuffle data

Indicates data shuffling between stages.

Memory and GC​

WARN MemoryStore: Not enough space to cache
INFO BlockManager: Block rdd_0_0 dropped from memory

Shows memory usage and garbage collection events.

Errors​

ERROR Executor: Exception in task 0.0 in stage 0.0
ERROR SparkUncaughtExceptionHandler: Uncaught exception

Highlights errors during execution.

Real-time Monitoring​

Live Log Streaming​

  • Logs update in real-time as Spark executes
  • Auto-scroll follows latest entries
  • Pause auto-scroll to read specific section
  • Click Resume to continue following

Monitoring Progress​

While cell executes:

  1. Watch Spark Logs for progress
  2. Monitor task completion
  3. Check for errors
  4. Review stage progression

Troubleshooting with Logs​

Job Stuck or Slow​

  1. Open Spark Logs
  2. Look for "Stage" messages
  3. Identify which stage is slow
  4. Review task execution logs
  5. Check for errors or exceptions

Out of Memory Errors​

Search for:

  • "OutOfMemoryError"
  • "not enough space to cache"
  • "Block dropped from memory"

Review memory configuration and data size.

Task Failures​

Look for:

  • "Exception in task"
  • "Task failed"
  • Executor error messages

Review error message for root cause.

Data Skew Issues​

Search for:

  • Imbalanced task execution times
  • Some executors finish quickly, others slow
  • Shuffle metrics showing uneven distribution

Network Issues​

Look for:

  • "Connection refused"
  • "Network timeout"
  • Shuffle read/write failures

Exporting Logs​

Save Logs to File​

  1. Click Export in Spark Logs panel
  2. Choose format:
    • Plain Text: Simple text file
    • CSV: Spreadsheet format
    • JSON: Structured format
  3. Save to local machine

Share Logs for Debugging​

  1. Export logs
  2. Share file with team
  3. Helps troubleshoot issues
  4. Include with error reports

Log Retention​

How Long Logs Retained​

  • Logs available while notebook session active
  • Historical logs available in Spark History Server
  • Logs cleared when notebook closed
  • Export for permanent storage

Accessing Historical Logs​

For completed jobs:

  1. Go to Jobs → Jobs Dashboard
  2. Click completed job
  3. View Spark History
  4. Access full historical logs

Performance Considerations​

Impact on Notebook​

  • Displaying logs adds minimal overhead
  • Filtering/searching logs is CPU intensive
  • Large log sizes may slow rendering
  • Consider exporting very large logs

Managing Log Size​

For very large log output:

  • Focus on ERROR level logs
  • Search for specific keywords
  • Export and analyze separately
  • Consider reducing RDD logging detail

Best Practices​

Log Monitoring​

✓ Do:

  • Monitor logs during first runs
  • Review errors immediately after cell
  • Search for specific issue indicators
  • Export important debugging sessions

✗ Don't:

  • Ignore warning messages
  • Run long jobs without monitoring
  • Forget to check ERROR level logs
  • Leave all log levels on for performance

Debugging Workflow​

  1. Run Spark cell
  2. Monitor Spark Logs for progress
  3. If error occurs:
    • Stop execution
    • Review error logs
    • Search for root cause
    • Fix code
    • Re-run

See Also​

  • [Create Notebook](./Create notebook.md)
  • [Editor Pane](./Editor Pane.md)
  • [Results Pane](./Results Pane.md)
  • Spark History Server