projects:commerce_st_chameleons:ghosttree_setup [wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Ghost Tree BeagleBones (unit1–unit5) ======

Summary of how the tree BeagleBone Blacks are set up (October 2026), with the low-level details needed to change or undo things in the field.

===== Overview =====

Each unit is a BeagleBone Black (Debian 10 Buster, BeagleBoard.org IoT image 2020-04-06, kernel 4.19) with a USB C-Media sound card and one LED strip.

Signal chain:

  mic -> USB card (hw:1,0) -> jackd -> Pd 0.55 (effectTry4.pd) -> speaker
                                         |
                                         +-- [netreceive] TCP 9990 --> ww-led --> OPC over UDP 7891 --> ledscape (opc-server) --> LEDs

  * **jackd** (''jackdaudio.service'') runs the USB sound card. The card only supports 44.1k/48k; never give jackd a lower ''-r''. The card takes 30–50 s to appear after boot.
  * **Pd** (''doTheTree.service'' -> ''~/bin/doTheTree.sh'') waits for jackd, sets the mixer, then runs ''~/ghostTree/effectTry4.pd''. The effect runs at 12 kHz inside subpatch ''main12k''. ''effectTry2.pd'' is the untouched 48k original; ''effectTry3.pd'' is the 12k version without recording.
  * **ww-led** (''ww-led.service'', ''~/dev/ww-led'') is a TCP client of Pd on 127.0.0.1:9990. It sends a frame to ledscape **only when Pd sends something**. Config is in ''python/config.py''; it drives one string of 100 LEDs. See [[#led_count_and_frame_rate|LED count and frame rate]].
  * **ledscape** (''ledscape.service'', opc-server, config ''/etc/ledscape-config.json'') plays its built-in "fade" demo after 5 s with no data. That idle fade is the intended look. opc-server always shows as a zombie in ''ps''; that's normal.
  * **jack-plumbing** (''jackPlumbing.service'' -> ''~/bin/runPlumbing.sh'', rules in ''/etc/jack-plumbing'') is kept for future use. It waits for jackd before starting.
  * **Mic recordings:** ''[pd recordLive]'' in effectTry4 writes the raw 48k mic to ''~/recordings/mic_YYYYMMDD_HHMMSS.wav'' while live.
  * **testPin** is still needed; only its log output is silenced.

^ Unit ^ IP ^ Notes ^
| unit1 | 10.42.0.61 | flasher image made from this unit |
| unit2 | (to do) | spare card ready (has ''rec/'') |
| unit3 | (to do) | spare card ready (has ''rec/'') |
| unit4 | 10.42.0.64 | was "unit2" on .62; reference unit; has ShellHub (''chameleons-bb'') |
| unit5 | (to do) | spare card ready (has ''rec/'') |

Login: ''debian'' with an SSH key. sudo asks for the debian password. Gateway 10.42.0.1, DNS 8.8.8.8 and 1.1.1.1.

===== Setting up the remaining units (unit2, unit3, unit5) =====

Everything is done from a PC by ''~/treeSetup/setup-tree.sh''.

==== What the PC needs ====

  * The whole ''treeSetup'' folder (about 80 MB): the five scripts, ''payload/'' and ''README.md''. Copy it to another PC as a tar archive (''tar -cf treeSetup.tar treeSetup''); a FAT USB drive loses the execute permissions Pd and the scripts need.
  * ''ssh'', ''scp'', ''ssh-copy-id'', ''ssh-keygen'', ''rsync'' 3.1 or newer, and an SSH key (''ssh-keygen -t ed25519'' if there is none).
  * Network access to the unit.
  * ''payload/'' is the copy of Pd, ghostTree, the ww-led edits, ''~/bin'' scripts and ''doTheTree.service'' taken from the reference unit. If you delete it, the script re-fetches it from unit4 (10.42.0.64, or set ''REF=ip'').

==== Steps per unit ====

  - Test the microSD card on any running unit before formatting it (this **erases** the card): <code>sudo bash ~/test-sdcard.sh</code> The result must be OK; fake cards fail here. On a PC with a card reader, use ''sudo bash test-sdcard.sh /dev/sdX'' from the treeSetup folder (find the device with ''lsblk''). It refuses non-removable disks and asks you to type yes.
  - Format it: <code>sudo bash ~/setup-sdcard.sh format</code>
  - Cards formatted before 2026-10-24 have no ''rec/'' folder. ''setup-tree.sh'' creates it on the inserted card; for a spare card, reformat it or run ''sudo bash ~/recordings-subfolder.sh'' with it inserted.
  - Put the card in the new unit, connect it to the network and power it up. Find its DHCP address (DHCP leases, ''ip neigh'', or ''connmanctl services'' on its console).
  - On the PC: <code>cd treeSetup && ./setup-tree.sh</code> It asks for:
    * the current address
    * the unit name (''unit1'' to ''unit5'')
    * the new static IP
    * netmask, gateway and DNS (Enter accepts the defaults)
  - Enter the debian password (first time, for ssh-copy-id) and the sudo password (once).
  - The script reboots the unit onto its new IP. It waits up to about 6.5 min for jack, Pd and ww-led, then prints a report. Check the report for:
    * all services ''active''
    * ''ww-led connected to Pd''
    * ''recordings: microSD card /dev/mmcblk0p1[/rec]''
    * ''fstrim.timer: enabled''
    * ''rsyslog inactive'', ''/tmp tmpfs'', ''commit=60''
    * ''RemoveIPC=no ok'' and ''fsck.repair=yes ok''
    * the crontab, and ''reboot rule ok''

The script stops if ''~/dev/ww-led'' on the unit is not at git commit ''400f44de...'', so it never overwrites an unknown version. The ww-led files it replaces are first saved as ''*.orig''.

==== What setup-tree.sh does ====

As debian:
  * Copies ''~/installs/pd-0.55-0'', ''~/ghostTree'' (without ''recordings''), the four ww-led files, and ''~/bin'' (''doTheTree.sh'', ''prune-recordings.sh'', ''restartIfJackIsJacked.sh'', ''runPlumbing.sh'').
  * Puts ''setup-sdcard.sh'', ''test-sdcard.sh'' and ''reduce-writes.sh'' in ''~''.
  * Adds two crontab lines:
    * ''*/10 * * * * /home/debian/bin/prune-recordings.sh''
    * ''*/5 * * * * /home/debian/bin/restartIfJackIsJacked.sh''

As root (''~/setup-tree/root-setup.sh''):
  * Sets the hostname and the ''127.0.1.1'' line in ''/etc/hosts''.
  * Purges Dataplicity (supervisor, ''/opt/dataplicity'', the dataplicity user).
  * Installs ''doTheTree.service'' and enables jackdaudio, doTheTree, ledscape, ww-led and jackPlumbing.
  * Disables the old jackTestLoop, jackStreaming and testAudio services.
  * Sets ledscape to 1 strip of 100 LEDs in ''/etc/ledscape-config.json'' (original kept as ''.orig'').
  * Runs ''reduce-writes.sh''.
  * Runs ''recordings-subfolder.sh'': the two fstab lines for the card, and ''rec/'' on the inserted card.
  * Enables ''fstrim.timer'' (weekly TRIM).
  * Adds ''fsck.repair=yes'' to ''/boot/uEnv.txt'', with a backup at ''/boot/uEnv.txt.bak''.
  * Writes ''/etc/systemd/logind.conf.d/keep-ipc.conf''.
  * Adds the sudoers reboot rule.
  * Sets the static IP with connman and reboots.

==== Alternative: start from the unit1 image ====

''~/chameleon-ghostTree-test-2026-10-23.img.xz'' (914 MB) is an eMMC flasher made with BeagleBoard's ''/opt/scripts/tools/eMMC/beaglebone-black-make-microSD-flasher-from-eMMC.sh''. It fits any card of 8 GB or more.

  - Write it to a card, replacing ''sdX'' with the card's device (check with ''lsblk''): <code>xzcat chameleon-ghostTree-test-2026-10-23.img.xz | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync</code>
  - Insert the card in the unit and power on. It rewrites the eMMC (10–20 min); the LEDs stop chasing when it's done. **Remove the card before the next boot**, or it flashes again.
  - **Always run ''setup-tree.sh'' on a flashed unit.** The flasher writes a brand-new minimal ''/etc/fstab'' onto the eMMC (''_generate_fstab'' in ''/opt/scripts/tools/eMMC/functions.sh''). A freshly flashed unit therefore has no card mount (recordings go to the eMMC), no ''commit=60'' and no ''/tmp'' in RAM. The image also predates the 2026-10-24 changes: 80/60 pruning, weekly TRIM and 100 LEDs. ''setup-tree.sh'' puts all of that back, so the image itself doesn't need updating for those.
  - The clone is a copy of unit1 on DHCP. Give it its own identity: <code>
sudo rm /etc/ssh/ssh_host_* && sudo dpkg-reconfigure openssh-server
sudo rm /etc/machine-id && sudo systemd-machine-id-setup
</code>
  - Run ''setup-tree.sh'' to set its name and IP.

How the image was made from the 116 GB card it was written to (shrunk so it fits small cards):
<code>
sudo umount /dev/sda1
sudo e2fsck -f /dev/sda1
sudo resize2fs /dev/sda1 4G
echo ',4G' | sudo sfdisk -N 1 /dev/sda
sudo dd if=/dev/sda bs=1M count=4100 status=progress | xz -T0 > chameleon-ghostTree-test-2026-10-23.img.xz
</code>

===== Field reference =====

==== Services ====

<code>
systemctl status jackdaudio jackPlumbing doTheTree ww-led ledscape
journalctl -u doTheTree -b          # Pd output for this boot
sudo systemctl restart doTheTree    # restart Pd only
sudo systemctl restart jackdaudio jackPlumbing doTheTree   # restart the audio chain
JACK_NO_START_SERVER=1 jack_lsp -c  # jack ports and connections
ss -tn | grep 9990                  # is ww-led connected to Pd?
</code>

Always set ''JACK_NO_START_SERVER=1'' when running jack tools by hand. Without it, a jack client can start its own temporary server when jackd isn't reachable.

==== Pd ====

Start command (in ''~/bin/doTheTree.sh''):
<code>
/home/debian/installs/pd-0.55-0/bin/pd -jack -nogui -send "sensitivity 35" -send "wet 0.6" -send "windowSize 2048" -send "micNoisegate 60" -send "exponent 2" -path /home/debian/ghostTree -path /home/debian/ghostTree/iemlib -stderr effectTry4.pd
</code>

  * Change the effect settings by editing the ''-send'' values, then ''sudo systemctl restart doTheTree''.
  * ''doTheTree.sh'' sets the volumes before starting Pd: ''amixer -c 1 -- sset Speaker 50%'' and ''amixer -c 1 -- sset Mic 100%''.
  * Keep ''-path /home/debian/ghostTree''. ELSE abstractions refer to ''else/xxx'' and won't load without it ("couldn't create").
  * ''doTheTree.service'' runs as debian with ''Restart=always'', ''RestartSec=5'' and ''After=jackdaudio.service''. Its ''LimitRTPRIO'' and ''LimitMEMLOCK'' are ''infinity''.

To run Pd by hand to see its messages:
<code>
sudo systemctl stop doTheTree
cd ~/ghostTree && /home/debian/installs/pd-0.55-0/bin/pd -jack -nogui ...same args... effectTry4.pd
sudo systemctl start doTheTree      # when done
</code>

==== LED count and frame rate ====

=== Number of LEDs (two places, must match) ===

  * **ww-led** decides the colours. ''trunk'' in ''~/dev/ww-led/python/config.py'' sets how many pixels each frame has. The range counts both ends, so ''[0, 99]'' is 100 LEDs.
  * **ledscape** drives the strip. ''ledsPerStrip'' in ''/etc/ledscape-config.json'' sets how many LEDs it actually outputs to, including the idle fade.
  * If ledscape is set lower, the extra LEDs stay dark. If ww-led is set lower, the extra LEDs only show the fade and don't react to Pd.

Show the current settings:
<code>
grep '^trunk' ~/dev/ww-led/python/config.py
grep -E 'ledsPerStrip|usedStripCount' /etc/ledscape-config.json
</code>

Set N LEDs (example: 100):
<code>
N=100
sed -i -E "s/^trunk = \[0, [0-9]+\]/trunk = [0, $((N-1))]/" ~/dev/ww-led/python/config.py
sudo sed -i -E "s/(\"ledsPerStrip\":\s*)[0-9]+/\1$N/" /etc/ledscape-config.json
sudo systemctl restart ledscape ww-led
</code>

''setup-tree.sh'' sets 100 on new units: ''LEDS=100'' in its root part and ''trunk'' in ''payload/ww-led/python/config.py''. The original ledscape config is kept as ''/etc/ledscape-config.json.orig''.

=== Frame rate ===

The rate the LEDs change is set by **Pd**, with a cap in ww-led. ledscape has no frame-rate setting; it smooths between frames itself (''enableInterpolation'').

  * **Pd** sends one ''amplitude'' message per envelope period. The period is half of ''windowSize'', and runs at 12 kHz inside ''main12k'': <code>messages per second = 12000 / (windowSize / 2)</code>
    * ''windowSize'' 2048 (current) gives about 11.7 per second.
    * 1024 gives about 23 per second.
    * 512 gives about 47 per second.
    * A smaller window also makes the amplitude follow the sound more nervously; a bigger one makes it smoother and slower.
    * Pd stops sending while the mic is below the noise gate (''micNoisegate''); the LEDs then go back to the fade after 5 s.
  * **ww-led** checks for new Pd data ''TARGET_FPS'' times per second (''~/dev/ww-led/python/config.py'', currently 25) and sends one frame when there is some. So it caps the rate at ''TARGET_FPS''; it never sends more frames than Pd sends messages.

Change Pd's rate (edit ''-send "windowSize 2048"'' in the Pd start line):
<code>
sed -i 's/-send "windowSize [0-9]*"/-send "windowSize 1024"/' ~/bin/doTheTree.sh
sudo systemctl restart doTheTree
</code>

Change ww-led's cap:
<code>
sed -i -E 's/^TARGET_FPS = [0-9]+/TARGET_FPS = 30/' ~/dev/ww-led/python/config.py
sudo systemctl restart ww-led
</code>

Show the current settings:
<code>
grep -o 'windowSize [0-9]*' ~/bin/doTheTree.sh
grep '^TARGET_FPS' ~/dev/ww-led/python/config.py
</code>

To change these on future units too, make the same edits in ''treeSetup/payload/bin/doTheTree.sh'' and ''treeSetup/payload/ww-led/python/config.py''.

==== Hostname ====

<code>
sudo sed -i "s/\bunit2\b/unit4/g" /etc/hosts     # old name -> new name in the 127.0.1.1 line
sudo hostnamectl set-hostname unit4
</code>
Change ''/etc/hosts'' first. Otherwise sudo prints "unable to resolve host" (harmless).

==== Static IP (connman) ====

<code>
connmanctl services          # shows e.g. ethernet_647060b02dc3_cable (the MAC without colons)
sudo connmanctl config ethernet_647060b02dc3_cable --ipv4 manual 10.42.0.64 255.255.255.0 10.42.0.1 --nameservers 8.8.8.8 1.1.1.1
sudo connmanctl config ethernet_647060b02dc3_cable --ipv4 dhcp     # back to DHCP
</code>
The change takes effect immediately, so an SSH session drops. Close it with Enter, then ''~.''. The setting is stored per MAC address, so a cloned eMMC on another board starts on DHCP.

==== Logging ====

Logs are kept in RAM only and are lost at reboot:
  * The systemd journal has no ''/var/log/journal'', so it is volatile.
  * rsyslog is disabled, so nothing goes to ''/var/log/syslog'' or ''/var/log/messages''.

Read the logs with ''journalctl'' (''-b'' for this boot, ''-u SERVICE'', ''-f'' to follow).

Persistent logging, for debugging something that happens across reboots:
<code>
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --list-boots          # after a reboot: previous boots are now there
journalctl -b -1                 # previous boot
</code>

Undo it (back to RAM only):
<code>
sudo rm -rf /var/log/journal && sudo systemctl restart systemd-journald
</code>

To bring back rsyslog and text files in ''/var/log'', use ''reduce-writes.sh undo'' (next section) or only:
<code>
sudo rm /etc/systemd/journald.conf.d/no-syslog.conf
sudo systemctl restart systemd-journald
sudo systemctl enable --now rsyslog
</code>

Files that survive reboots anyway:
  * ''~/jack-reboots.log'': every reboot made by the jack watchdog, with the reason.
  * The recordings.

==== eMMC write reduction (reduce-writes.sh) ====

<code>
sudo bash ~/reduce-writes.sh        # apply (setup-tree does this)
sudo bash ~/reduce-writes.sh undo   # put everything back
</code>

What it changes:
  * testPin output is dropped with ''/etc/systemd/system/testPin.service.d/quiet.conf'' (''StandardOutput=null''). The service still runs.
  * journald stops forwarding to syslog (''/etc/systemd/journald.conf.d/no-syslog.conf''), and rsyslog is disabled.
  * Root filesystem ''commit=60'' in ''/etc/fstab''. Data reaches the eMMC every 60 s instead of every 5 s, so a power cut can lose up to the last minute of writes.
  * ''/tmp'' is in RAM: ''tmpfs  /tmp  tmpfs  defaults,noatime,nosuid,nodev,mode=1777,size=16M  0  0''. If it fills, writes to /tmp fail with "No space left"; nothing is cleaned automatically. Only tiny files live there (for example ''/tmp/jack-unreachable-count'').

After ''undo'', ''/tmp'' returns to disk at the next reboot.

Possible later step: ''overlayroot'' (read-only root with a RAM overlay). The rcn-ee package exists for Buster, and the kernel and initramfs support it. It's not installed.

==== Power loss ====

  * ''fsck.repair=yes'' is on the ''cmdline='' line of ''/boot/uEnv.txt''. After a power cut, boot repairs the filesystem itself instead of stopping at an emergency shell.
    * Check it's active: ''grep fsck.repair /proc/cmdline''
    * Backup: ''/boot/uEnv.txt.bak''. On unit1 and unit4 the backup is ''~/uEnv.txt.bak'', because that change was made by hand.
    * To edit uEnv.txt as debian, use ''sed ... > ~/uEnv.new && cat ~/uEnv.new > /boot/uEnv.txt''. ''sed -i'' can't create its temp file in /boot.
  * ext4 has a journal, so a power cut loses at most the last ~60 s of writes (commit=60) and doesn't corrupt the filesystem.

==== jack: RemoveIPC fix ====

''/etc/systemd/logind.conf.d/keep-ipc.conf'':
<code>
[Login]
RemoveIPC=no
</code>
By default, when the last login session of user debian ends (for example an SSH logout), systemd-logind deletes all shared memory and semaphores owned by debian. jackd also runs as debian, so this broke jack: "JACK semaphore error", "jack_shm_lock_registry fails", and no sound. Never remove this file. It needs a reboot to take effect.

==== Jack watchdog ====

''~/bin/restartIfJackIsJacked.sh'' runs from debian's crontab every 5 min. It reboots the unit if:
  * jackd is above 80% CPU, or
  * jackd is running but ''jack_lsp'' can't reach it on 2 checks in a row.

It logs the reason to ''~/jack-reboots.log''. To reboot without a password it relies on ''/etc/sudoers.d/reboot-debian'':
<code>
debian ALL=(root) NOPASSWD: /sbin/reboot
</code>

To pause it while debugging (otherwise stopping jackd by hand can get the unit rebooted):
<code>
crontab -e       # put # in front of the restartIfJackIsJacked.sh line; remove it when done
</code>

==== Recordings and microSD cards ====

  * The card is ext4, labelled ''recordings''. The recordings are in its ''rec/'' folder, never in its top folder.
  * **Why the subfolder:** at every boot, the boot loader (U-Boot 2019.04) checks the card's top folder for boot files. Once that folder holds a few hundred files, U-Boot hangs: only the power LED lights, the user LEDs never do, and the unit boots fine without the card. Tested on unit1 on 2026-10-24 with the same card: 47 files in the top folder boots, 547 hangs, and 3000 files in ''rec/'' boots.
  * Two ''/etc/fstab'' lines mount the card on ''/mnt/sdcard'', then its ''rec/'' folder on ''/home/debian/recordings'' (a bind mount): <code>
LABEL=recordings  /mnt/sdcard  ext4  noatime,commit=60,nofail,x-systemd.device-timeout=120s  0  2
/mnt/sdcard/rec  /home/debian/recordings  none  bind,nofail,x-systemd.requires-mounts-for=/mnt/sdcard,x-systemd.before=doTheTree.service  0  0
</code>
  * ''~/recordings-subfolder.sh'' (run with sudo) sets this up on a unit. It writes both lines, keeping the old fstab as ''/etc/fstab.before-subfolder'', and creates ''rec/'' on the inserted card. If the card is mounted the old way, it also moves the card's recordings into ''rec/''. A card formatted before this change has no ''rec/'' and is ignored (recordings go to the eMMC) until this script has been run with it inserted.
  * The card is mounted **only at boot**. Inserting a card into a running unit does nothing.
  * With no card at boot, neither mount happens and the ''~/recordings'' folder on the eMMC is used. Pd always writes to the same path and doesn't know which one it is. Pd then starts about 2 minutes after boot, because of the 120 s card timeout.
  * The 120 s timeout is needed because udev only creates the label link about 60 s into boot.
  * ''nofail'' means the unit boots normally without a card.
  * Check where recordings are going: <code>
findmnt ~/recordings          # SOURCE /dev/mmcblk0p1[/rec] = card; no output = eMMC
ls -A /mnt/sdcard             # card top folder: should only hold lost+found and rec
</code>
  * Swap a card on a running unit:
    - ''sudo umount ~/recordings /mnt/sdcard''
    - Swap the card.
    - ''sudo mount /mnt/sdcard && sudo mount ~/recordings'', or reboot.
    - While no card is mounted, recordings go to the eMMC.
  * **Pruning** (''~/bin/prune-recordings.sh'', every 10 min) deletes the oldest ''mic_*.wav'' files:
    * on the card: once it is more than 80% full (''HIGH_PCT''), it deletes the oldest recordings in one batch until the card is down to 60% (''LOW_PCT'');
    * on the eMMC, until the folder is under 2 GB (''MAX_MB'').
  * **TRIM:** ''fstrim.timer'' runs ''fstrim'' weekly. It tells the card and the eMMC which blocks are free, so their wear leveling can use them. Without TRIM, space freed by deleting files still looks used to the card. Check with ''systemctl status fstrim.timer''; run it by hand with ''sudo fstrim -av''.
  * ''test-sdcard.sh'' (destructive) writes markers every 512 MB across the card and reads them back, which catches fake-capacity cards (one "NCard" 32 GB card failed).
  * ''setup-sdcard.sh'' (destructive) creates one ext4 partition labelled ''recordings''.
    * It creates the ''rec/'' folder on the card.
    * With no argument, it also moves the existing eMMC recordings into ''rec/'' and writes the two fstab lines.
    * With ''format'', it only formats the card.
    * It only ever touches ''/dev/mmcblk0'' (the SD slot), and refuses if the system is booted from SD.

==== Dataplicity ====

Removed from all units (supervisor purged, ''/opt/dataplicity'' and the dataplicity user deleted). Nothing to undo.

==== ShellHub (unit4 only) ====

  * The agent is a native armv7 binary, not Docker: ''/usr/local/bin/shellhub-agent'', settings in ''/etc/shellhub/agent.env'' (server, tenant ID, key ''/etc/shellhub/agent.key'', ''PREFERRED_HOSTNAME=chameleons-bb''), run by ''shellhub-agent.service''.
  * It was built with ''GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=0'', with ''passwordHandler'' in ''agent/server/authentication.go'' changed to return false, so **password login is refused** and only keys work. The built-in updater does nothing; update by rebuilding and copying the binary.
  * Remove from a unit: <code>sudo systemctl disable --now shellhub-agent</code>

==== Other gotchas ====

  * **Kubuntu remembers SSH passwords.** ''ksshaskpass'' saves them in KWallet. If one was typed wrong, delete it in KWallet Manager under kdewallet -> ksshaskpass. To be asked every time for one login: ''SSH_ASKPASS_REQUIRE=never ssh ...''.
  * **Host key warnings.** After moving or reflashing a unit, the PC's ''known_hosts'' has the wrong key. Clear it with ''ssh-keygen -R 10.42.0.xx''. Only do this when you know why it changed.
  * **IP clashes.** Keep other devices (for example the MikroTik mAP lite) off the units' addresses.