(r2) PatasSpark < Main

Tags: view all tags
---+ Spark on the CLMS cluster

Spark jobs may be submitted from any node; the master URL, if you need it, is =spark://diana.local:7077=, although most spark commands will pick this up automatically.

The job queue and node status may be monitored via the Spark Master's web interface, http://diana.ling.washington.edu:8080/.

General information on Spark, along with examples, can be found on the [[http://spark.apache.org/][Apache Spark homepage]].

There are also example jobs in =/opt/spark/examples=. For example, to run the Pi approximation example with 1000 rounds, you would type
<pre><verbatim style="background-color: transparent;">spark-submit /opt/spark/examples/src/main/python/pi.py 1000</verbatim>
</pre>

This will produce a lot of job information output, which may bury the output of the actual job, so you may want to split up stdout and stderr like so:
<pre><verbatim>spark-submit /opt/spark/examples/src/main/python/pi.py 1000 >pi.out 2>pi.err
</verbatim>
</pre>

---++ Hadoop File System

The files are not guaranteed to be spread across the cluster for use by Spark. A workaround is to place it in the HDFS:

hadoop fs -put local_dir_name my_dir

You can verify its contents like so:

hadoop fs -ls my_dir

And access it like so:

"hdfs://diana.local:9000/user/" + "username/my_dir"

The uri shown in ls does not work:

"hdfs://" + /user/jcadigan/en_ar_data/fr_eng_1M.eng.true
Topic revision: r2 - 2016-08-21 - jcadigan
~~Edit~~
~~Attach~~