(r1) PatasSpark < Main

Tags: view all tags
---+ Spark on the CLMS cluster

Spark jobs may be submitted from any node; the master URL, if you need it, is =spark://diana.local:7077=, although most spark commands will pick this up automatically.

The job queue and node status may be monitored via the Spark Master's web interface, http://diana.ling.washington.edu:8080/.

General information on Spark, along with examples, can be found on the [[http://spark.apache.org/][Apache Spark homepage]].

There are also example jobs in =/opt/spark/examples=. For example, to run the Pi approximation example with 1000 rounds, you would type
<pre><verbatim style="background-color: transparent;">spark-submit /opt/spark/examples/src/main/python/pi.py 1000</verbatim>
</pre>

This will produce a lot of job information output, which may bury the output of the actual job, so you may want to split up stdout and stderr like so:
<pre><verbatim>spark-submit /opt/spark/examples/src/main/python/pi.py 1000 >pi.out 2>pi.err</verbatim>
</pre>
Topic revision: r1 - 2015-03-25 - brodbd
~~Edit~~
~~Attach~~