Spark on the CLMS cluster

Spark jobs may be submitted from any node; the master URL, if you need it, is spark://diana.local:7077, although most spark commands will pick this up automatically.

The job queue and node status may be monitored via the Spark Master's web interface, http://diana.ling.washington.edu:8080/.

General information on Spark, along with examples, can be found on the Apache Spark homepage.

There are also example jobs in /opt/spark/examples. For example, to run the Pi approximation example with 1000 rounds, you would type

spark-submit /opt/spark/examples/src/main/python/pi.py 1000

This will produce a lot of job information output, which may bury the output of the actual job, so you may want to split up stdout and stderr like so:

spark-submit /opt/spark/examples/src/main/python/pi.py 1000 >pi.out 2>pi.err