Matter and Void

Raspberry Pi Timelapse and upload to YouTube

May 17, 2019project

I played with creating time lapse videos with the pi a few years ago, but recently came up with the idea of uploading a daily video to YouTube.

This has been running since late January 2019. Here’s one of the uploads:

https://www.youtube.com/watch?v=2Kc-LLQsCiU

This post describes the bash scripts to automate this process.

The goal was to create a timelapse video once every day, upload that video to YouTube and then delete the local video, this way I wouldn’t have to deal with storing the video files myself.

The following are a few BaSH scripts I put together to perform the tasks.

I had a gigabyte brix computer, so decided to use that for encoding and uploading to not overload the pi in case that interfered with taking photos while it was encoding.

The Pi and the gigabyte are on the same LAN. I set a static IP address for the gigabyte in the router configuration so there wouldn’t be a risk of it getting assigned a different IP address over time.

The high level tasks are:

On the pi:

  1. Take a photo every 30 seconds and store it locally.
  2. Rsync the photo to a separate machine which will encode the photos into a video once a day.

On the gigabyte:

  1. A directory of photos per day (YYYY-MM-DD) is created to be copied from the Pi.
  2. Cron task running once a day encodes yesterday’s photos into a video.
  3. The resulting video file is uploaded to YouTube.
  4. On successful upload, yesterday’s photos are deleted.
  5. Once a month the previous month’s video files are deleted from the filesystem.

This assumes some first time setup for the Pi (also assumes model A or B):

  • Wifi USB card.
  • Camera is plugged in (blue tape facing ethernet port).

Here are the steps, based on the UK raspbian images.

US setup

Usually the problem is the ’|’ character. if you type this and get ’~’ then you should update the keyboard layout:

sudo raspi-config

-> 4. Localization options

-> Change Keyboard Layout

-> Wait a bit to reload the keymaps.

-> other -> generic 104 key, not the International one.

It will ask you to pick from some UK options, choose other

English (US)

Then go through the next few screens

Network

Via raspi-config

sudo raspi-config

-> 2. Network options, you must know your SSID and passphrase to connect to.

Via config files

Wireless connection info in /etc/wpa_supplicant/wpa_supplicant.conf

This file is written when you use the raspi-config setup above.

If all else fails, this page has instructions on how to use wpa_supplicant:

https://shapeshed.com/linux-wifi/

Timezone

For list of timezones:

ls /usr/share/zoneinfo/
# For US timezones:
ls /usr/share/zoneinfo/US

sudo vi /etc/timezone insert your timezone e.g.:

America/New_York

and you can also update the link

sudo rm /etc/localtime
sudo ln -s /usr/share/zoneinfo/US/Eastern /etc/localtime

check your time with date.

Camera

sudo raspi-config

Interfacing options

SSH

# This makes it start on boot:
sudo systemctl enable ssh

# This starts it now:
sudo systemctl start ssh

Auth is password based by default. (Adjust based on your level of paranoia.)

Back to the timelapse scripts.

With all that setup done, on the Pi in the home directory:

mkdir timelapse

and a script do_timelapse.sh in home dir:

#!/bin/bash -

mode=auto
sleep_for=30
# Change per your setup
video_host=192.168.1.172
# Change per your setup
video_user=user
# Change per your setup
video_dir=timelapse/front_window
# You can play with resolution to your liking.
width=3280
height=2464

while true; do
  output_dir=$(date +"%F")
  mkdir -p $output_dir

  current_hour=$(date +"%H")
  # ${var#0} syntax removes leading zeros for integer comparison.
  if [[ "${current_hour#0}" -gt 21 || "${current_hour#0}" -lt 9 ]]; then
    mode=night
  fi
  current_datetime=$(date +"%FT%H-%M-%S%Z")
  output_filename="$output_dir/${current_datetime}.jpg"
  raspistill -w "$width" -h "$height" -ex "$mode" --nopreview -o "$output_filename"

  ssh -i /home/pi/.ssh/gigabyte "$video_user"@"$video_host" "mkdir -p ${video_dir}/$output_dir"
  rsync -avhz -e "ssh -i /home/pi/.ssh/gigabyte" "$output_filename" "$video_user"@"$video_host":"${video_dir}/$output_filename"
  rm "$output_filename"
  sleep $sleep_for
done

I played with the ‘night’ mode and the exposure comes out better at night.

An improvement to this would be to contact a web service to request when daylight hours are based on the timezone of the OS and use that instead of hard-coding the hours.

This setup requires creating an SSH keypair on the Pi and adding the public key to the authorized_keys file on the remote machine to allow automated copying of files between machines.

To generate the keypair (run on the pi):

ssh-keygen -t rsa -b 4096 -C comment -f ~/.ssh/key_name
# Copy the key to authorized_keys using one of these methods:
ssh-copy-id -i ~/.ssh/keyname user@IP_ADDRESS
# or
cat "$identity_file.pub" | ssh $user@$host 'cat >> ~/.ssh/authorized_keys'

Confirm SSHing after that. You should not be prompted for a password.

You could also improve on error handling: if rsyncing fails, for example, you probably don’t want to delete the image file so you can recover it later. You could also email an address to notify of the failure.

Put the following in: /etc/systemd/system/timelapse.service:

[Unit]
Description=Take some photos
After=multi-user.target

[Service]
Type=simple
ExecStart=/home/pi/timelapse/do_timelapse.sh

[Install]
WantedBy=multi-user.target

Enable the timelapse script on boot:

sudo systemctl enable timelapse

enable right now with:

sudo systemctl start timelapse

This unit file has the effect of starting the service on boot as well as wiring output to syslog. Any output to standard out will visible in the /var/log/syslog{*} files with output like so:

May 17 13:14:52 raspberrypi do_timelapse.sh[20637]: sending incremental file list

On the encoding and uploading machine

mkdir timelapse

In the timelapse directory put the following script as run_make_movie.sh:

#!/bin/bash -
#
# Lists directories of current year in directory "$movie_dir", picks
# the most recent one as sorted by date, and encodes a movie from the image stills.
# Then uploads the movie to youtube. If uploaded successfully, then deletes the stills.
#

echo "running run_make_video script"
home_dir="/home/user/timelapse"
movie_dir="front_window"

pushd "${home_dir}/${movie_dir}"

year=$(date +%Y)

# Get yesterday's directory
pics_dir=$(ls | grep "^$year" | sort -n | tail -n 2 | sort -nr | tail -n 1)

echo "pics dir: $pics_dir"

if [ -z "$pics_dir" ]; then
  echo "$pics_dir is missing."
  exit 1
fi

echo "./make_video.sh $pics_dir"
./make_video.sh "$pics_dir"

movie_filename="timelapse_${pics_dir}.avi"
movie_title="$pics_dir My video description timelapse"

popd

pushd "${home_dir}"

echo "./upload_file.sh ${movie_dir}/${movie_filename} \"$movie_title\""
./upload_file.sh "${home_dir}/${movie_dir}/${movie_filename}" "$movie_title"

if [ $? -eq 0 ]; then
  echo "Successfully uploaded ${movie_filename} to Youtube"

  echo "rm -rf ${home_dir}/${movie_dir}/$pics_dir"
  rm -rf "${home_dir}/${movie_dir}/$pics_dir"
else
  # To setup mail:
  # https://rianjs.net/2013/08/send-email-from-linux-server-using-gmail-and-ubuntu-two-factor-authentication  
  echo "For some reason the youtube uploader script failed." | mail -s "Your app YT-Uploader FAILED" "youremail@mail.com"
fi

# Clean up previous month's videos if they exist

if command ls "${home_dir}/${movie_dir}/timelapse_$(date --date="1 month ago" +%Y-%m)"* &> /dev/null; then
  last_month_vids=$(command ls "${home_dir}/${movie_dir}/timelapse_$(date --date="1 month ago" +%Y-%m)"*)
  echo "removing last month's videos: $last_month_vids"
  rm -f "$last_month_vids"
fi

popd

This is the main runner script, it relies on two other scripts: make_video.sh and upload_file.sh.

makevideo.sh goes in `/home/user/timelapse/$projectname` make_video.sh:

#!/bin/bash -
#
# Encode a directory of images into a movie file.
# Intended use is to make a timelapse movie.
#
set -eu

movie_width=3280
movie_height=2464

usage() {
  echo
  echo "Encode a directory of images into a movie."
  echo "$0 <subdirectory>"
  echo
}

if [ $# -eq 0 ]; then
  echo 'Missing subdirectory argument. Exiting'
  usage
  exit 1
fi

if [ ! -d "$1" ]; then
  echo
  echo "Directory \"$1\" does not exist. Exiting."
  echo
  exit 2
fi

subdir="$1"
# Remove trailing slash of filename
# same effect as subdir=$(echo "$1" | sed s:/::g)
subdir=$(sed s:/::g <(echo "$1"))

if [ ! -d "$subdir" ]; then
  echo
  echo "Directory \"$subdir\" does not exist. Exiting."
  echo
  exit 2
fi

echo "subdir: $subdir"

filelist="stills.txt"

rm -f "$filelist"

# awk -v is used to make a bash variable available in the awk script.
# This creates lines in $filelist like: 2019-05-16/2019-05-16T23-25-14EDT.jpg
# because the image files are in the subdirectory 2019-05-16.
ls "$subdir" | awk -v dir="$subdir" '{print dir"/"$1}' > "$filelist"

echo 'wrote file list:'
head "$filelist"

movie_file="timelapse_${subdir}.avi"

mencoder \
  -nosound \
  -ovc lavc \
  -lavcopts vcodec=mpeg4:mbd=2:trell:autoaspect:vqscale=3 \
  -vf scale="$movie_width":"$movie_height"\
  -mf type=jpeg:fps=20 mf://@"$filelist" \
  -o "$movie_file"

I had to play around with the mencoder config, copied from somewhere on the Net, -ovc is to set the output video codec. -vf is to set a video filter I’m honestly not sure what the -lavcopts options after setting the video codec to mpeg4 do… :)

After this runs we now have a video file which we can now upload to YouTube:

To actually upload the video file I am using this go package: https://github.com/porjo/youtubeuploader

You can download a binary from the releases page and follow the instructions to deal with authenticating with YouTube.

The upload script assumes the binary exists in /home/user/timelapse/youtubeuploader_linux_amd64

And for the upload script itself, add the following to /home/user/timelapse/upload_file.sh:

#!/bin/bash -

set -eu

usage() {
  echo
  echo "Upload a movie file to youtube."
  echo "$0 <filepath> <movie title>"
  echo
}

if [ $# -eq 0 ]; then
  echo 'Missing filepath and movie title argument. Exiting'
  usage
  exit 1
fi

if [ $# -eq 1 ]; then
  echo 'Missing movie title argument. Exiting'
  usage
  exit 2
fi

if [ ! -f "$1" ]; then
  echo
  echo "File \"$1\" does not exist. Exiting."
  echo
  exit 3
fi

video_filename="$1"
video_title="$2"

./youtubeuploader_linux_amd64 \
  -filename "$video_filename" \
  -privacy public \
  -title "$video_title" \
  -description "" \
  -headlessAuth

Then the last step is to add a cron job to run the upload script each day:

crontab -e

Add the following contents:

# This runs every day of the month, every month, on every day of the week at the
# first minute start of second hour of the day.
0 1 * * * /home/user/timelapse/run_make_movie.sh > /home/user/timelapse_run_make_video_out.txt 2>&1

It took me a bit of manual running of the scripts and incrementally editing to get everything working so don’t be surprised if you’ll need to do the same.


Dan Vingo
This is the personal site of Dan VingoAbout