Setting up a cluster of Raspberry Pi's

My mini-computer is basically a cluster of small computers running specialized software that I wrote.

I saw a lot of basic information on setting up a cluster but no one really gave details. So I will.

I'll add to this as needed, but here is a start:

############### Setting up a Cluster ###############

This is assuming you are using a set of raspberry-pi computers. I use
the current (2/20/2016) operating system Jessie for the Host computer,
and Jessie-lite for the Client computers. The difference is Jessie-lite
does not have a lot of un-needed packages and since the are headless, they
also do not need x-windows etc.

NOTE: it stopped working for me on or about 10/2016. I found I needed to add the package: mpi-default-bin on all processors except my primary. Even after I did the sudo apt-get update.

In addition, the structure of the machine file changed as did the command syntax. read the man pages.


You will have at a minimum of two computers: the Host and a Client.



#########################
#    To add the Host:   #
#########################

Physically connect power, wifi (if used) and Ethernet to a switch.
Add the following to /etc/rc.local:
iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT
sleep 5
# get the rpc services started
service rpcbind restart
# and make sure the nfs services start
service nfs-kernel-server restart


I set up /etc/dhcpcd.conf as follows: (all this is after the boilerplate)
interface eth0
inform 172.31.4.10

interface wlan0
static routers=192.168.1.1
SSID NETGEAR49
inform 192.168.1.70

interface wlan1
SSID NETGEAR
inform 192.168.1.71
static routers=192.168.1.1


Add the following to /etc/exports:
/media/pi/bfe7b4b9-cef9-411e-a7bd-38a9b3f93150/pi/Shared 172.31.4.11(rw,sync,no_subtree_check,no_root_squash)
/media/pi/bfe7b4b9-cef9-411e-a7bd-38a9b3f93150/pi/Shared 172.31.4.12(rw,sync,no_subtree_check,no_root_squash)
/media/pi/bfe7b4b9-cef9-411e-a7bd-38a9b3f93150/pi/Shared 172.31.4.13(rw,sync,no_subtree_check,no_root_squash)
/media/pi/bfe7b4b9-cef9-411e-a7bd-38a9b3f93150/pi/Shared 172.31.4.14(rw,sync,no_subtree_check,no_root_squash)

(This assumes /media/pi/xxx/Shared is where you want your nfs directory)
The 172.x.x.x is the ip addresses of the other Client computers.

Now edit /etc/hosts.allow:
ALL: 172.31.4.11 ALL
ALL: 172.31.4.12 ALL
ALL: 172.31.4.13 ALL
ALL: 172.31.4.14 ALL
ALL: 172.31.4.15 ALL

Now reboot and the Host should be set up correctly.
From the client: showmount -e pi1 will show all nfs mount points on pi1

##########################
#   To add a Client:     #
##########################

Physically connect power and ethernet. Install the operating
software. Turn on:

At new device 1st turn-on run: sudu raspi-config
Expand the file system
Set timezone
Set keyboard setup
Allow ssh to run
Reboot

Edit /etc/rc.local:
sleep 5
ifconfig eth0 172.31.4.12
route add default gw 172.31.4.10

# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi

# connect shared NFS mount
mount pi1:/media/pi/bfe7b4b9-cef9-411e-a7bd-38a9b3f93150/pi/Shared /mnt/nfs/Share

exit 0

Now create the nfs share directory:
mkdir /mnt/nfs
mkdir /mnt/nfs/Share

Edit /etc/resolvconf.conf:
resolv_conf=/etc/resolv.conf
# If you run a local name server, you should uncomment the below line and
# configure your subscribers configuration files below.
#name_servers=127.0.0.1
name_servers=8.8.8.8
name_servers-8.8.4.4

# Mirror the Debian package defaults for the below resolvers
# so that resolvconf integrates seamlessly.
dnsmasq_resolv=/var/run/dnsmasq/resolv.conf
pdnsd_conf=/etc/pdnsd.conf
unbound_conf=/var/cache/unbound/resolvconf_resolvers.conf


Edit /etc/resolv.conf:
# Generated by resolvconf
nameserver 8.8.8.8
nameserver 8.8.4.4

Set the computers name with: sudo vi /etc/hostname
pi6

Reboot

Now from your host computer you should be able to ssh into your new node.
Edit the /etc/hosts file to show all nodes on the system:
192.168.1.70    kurts_office

127.0.1.1    raspberrypi
172.31.4.10     pi1
172.31.4.11     pi2
172.31.4.12     pi3
172.31.4.13     pi4
172.31.4.14     pi5

Verify nfs is set up OK by looking to files in /mnt/nfs/Share

From the Host computer, copy the ssh files needed to avoid using a password:
ssh-copy-id pi6

Then ssh into the new computer to verify it copied correctly.

If using mpich use apt-get install to install the libraries and files:
sudo apt-get install libmpich-dev
sudo apt-get install libmpich12:armhf

At this point all needed files are probably installed. But lets make sure:
sudo apt-get install libmpl-dev
sudo apt-get install libmpl1:armhf
sudo apt-get install libopa-dev
sudo apt-get install libopa1:armhf
sudo apt-get install mpich
sudo apt-get install mpich-doc
sudo apt-get-install mpi-default-bin (new 10/2016)

####################
####################

I add a few files on all machines. I use scp to update from
one master copy on the Host.

from ~/.bash_aliases:
alias vi='vim.tiny'
alias l='ls -FhlAp'
alias df='df -h'
alias cls='clear'
#alias ps='ps -eLfP'
alias ps='ps  -c -f afSU pi --sort=uid,+pid'
alias free='free -h'
alias dir='ls -ld */'
alias sha1='sha1sum'
alias ll='ls -Fhlp'
alias freq='cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq'
alias temp='cat /sys/class/thermal/thermal_zone0/temp'

scp /home/pi/.bash_aliases pi2:/home/pi/.bash_aliases
scp /home/pi/.bash_aliases pi3:/home/pi/.bash_aliases
scp /home/pi/.bash_aliases pi4:/home/pi/.bash_aliases
scp /home/pi/.bash_aliases pi5:/home/pi/.bash_aliases

Since you are copying user files, you do not need to run scp as sudo.


The default debian install has an issue with rsyslog:

edit the last few lines of /etc/rsyslog.conf
to remove to daemon log recording then restart the process:

sudo service rsyslog restart