Schedule stream recordings from the command line, Part 1

"Radio continues to evolve in the digital age." It's also the most inviting medium when it comes to tinkering with technology. With World Radio Day 2014 coming up on February 13, I would like to share a hack I developed to help radio-makers discover and utilize rare content.

I've loved radio ever since I built my first crystal radio receiver as a kid with my dad. Radio has changed a lot since then, so has the way I listen. I used to do a lot of taping at home. Now I'm a podcast addict, and for me that involves hunting for and collecting spoken-word radio jewels that are not included in any podcast.

To do this, I wanted a script to schedule recordings for web streams, one that could run on my server and not my desktop machine (because I don't want my desktop switched on all the time). The result is streamplan.sh, a bash script that can be started from the command line.

I wrote this post in two parts, stay tuned for part two in the coming days:

• Which application to use for recording live streams, and why
• How to configure and schedule stream recording with streamplan.sh

Which application to use for recording live streams, and why

The first step in building a scheduler for stream recording is researching what recorder to use. This post is dedicated to the comparison of various options. The next post will go into more detail regarding the configuration and scheduling part. The recording app would need to...

• ... Be controlled from the command line. I want this app to run on my server and that I access through the terminal.
• ... Transcode files. Streams come in different formats like mp3, ogg, windows media and more. I want to transcode all into mp3 or ogg.
• ... Be told to stop after a certain time. The player should kill itself and stop recording. Otherwise the hard disk would fill up in case an external kill signal didn't come.
• ... Handle constant and variable bitrate, because both systems are used in popular streams.
• ... Read playlists as well as mount points, because many links to live streams are actually links to playlists. More about this in relation to avconv / ffmpeg below.

Many media players can record streams, and there are a number of applications dedicated to stream ripping. Based on the criteria above, I shortlisted a couple:

Streamripper: Recording tracks like sliced bread

Streamripper is great when it comes to converting music streams into single files. It records the stream, chops it up, changes the ID3 tags according to the stream information, and saves individual files in a separate folder. Since I mainly record larger shows in one piece, I added options on the command line to not add a directory for each stream (-s) and rip to a single file (-a followed by the filename) into a specified directory (-d followed by the directory) and it would terminate automatically after a given time (-l followed by the duration in seconds):

streamripper http://link.to/stream -d /path/to/ -a localfile.mp3 -s -l duration

But Streamripper did not work for some mp3 streams. I assume it cannot handle variable bitrate and/or not extract mount points from playlist; the Streamripper tutorial mentions this. When I tried to record mp3 streams, it worked with an NPR stream, but not with Dradio.de which is encoded in a variable bitrate.

MPlayer: The player that records

I had used MPlayer before but I didn't know it is also very good for grabbing live streams. Simply add the option -dumpstream, tell it what file to record to (-dumpfile followed by the filename) and the stream to listen to (-playlist followed by the stream url):

mplayer -dumpstream -dumpfile "/path/to/localfile.mp3" 

-playlist "http://link.to/stream"

To the best of my knowledge, transcoding happens automatically. The file ending of the dumpfile will specify what format to transcode to (e.g. .mp3).

But that the way in which Mplayer can be told to stop after a certain time (using the option -endpos) works only for static files, not for streams — which is what this blog post is about.

Avconv (aka ffmpeg): The Swiss army knife with an Achilles heel

Avconv is a command line tool for transcoding multimedia files. It is part of a fork of the widely known ffmpeg. When I began doing this research, I was certain that I would end up working with avconv. And if you take a look at their documentation page, you get the impression that avconv can do pretty much everything.

But I immediately ran into problems recording my sample streams. It took me some time to understand what was actually happening (or rather not happening). Many links to web streams are actually links to playlist files, not to the mount point of the stream itself. To give you an example, try running these two commands in your terminal window:

wget http://www.dradio.de/streaming/dlf_hq_ogg.m3u
cat dlf_hq_ogg.m3u

The first line retrieves the content of the file, which is the link to the live stream of dradio.de. You might expect this to be the audio stream, but it is not. It is actually a text file that contains a single line - which points to another URL. You can see this URL using the second command which spits out the content of the linked file, the link to the actual stream.

Try again with another example, NPR:

wget http://www.npr.org/streams/mp3/nprlive24.pls
cat nprlive24.pls

This file contains metadata and a link to the actual stream in row three:

[playlist]
NumberOfEntries=1
File1=http://nprdmp.ic.llnwd.net/stream/nprdmp_live01_mp3
Title1=NPR 24 Hour Program Stream

There are many reasons for using a playlist file instead of pointing directly to the stream. It allows you to change the streaming server in the background without having to change the links to the stream. You can also play one or more static files (e.g. advertisement or jingles) before the live stream starts. And, as is the case in the NPR example, you can add metadata for the player. You can also include a fallback stream as the second URL in the playlist.

Most players will take the file, read it and then start playing the content. Not avconv. It requires the actual stream and cannot interpret playlists.

VLC : My choice for recording live streams

Everybody seems to know the VLC player. Many people also use it to rip media and some to transcode files from one format to another. VLC does everything on my checklist, but there is one little problem.

VLC runs from the command line with the command "cvlc". The player transcodes files, it also handles constant and variable bitrates. VLC also correctly reads and interprets playlists, the one feature that avconv fell short on.

But you will remember that I needed process that could kill itself before filling up the hard disk. And while VLC can do this, it behaved strangely when I tested it.

The following command was suggested in the VLC forum to stop the recording automatically after $LENGTH seconds:

cvlc "http://link.to/stream" --sout file/mp3:/path/to/localfile.mp3 --run-time 

$LENGTH --stop-time=$LENGTH vlc://quit

I tried it in different combinations, using only run-time or only stop-time or both, but it didn't work as I expected. Instead, the command runs twice and then stops. For example, I want to record for one hour ($LENGTH in seconds = 3600) at 8pm. Acting on the command above, VLC records from 8 to 9pm, then stops. Then it starts recording again, from 9 to 10pm, overwriting the first recording.

Is it just me? If you know what I am doing wrong here, please comment. I am happy to learn.

Now the good news: VLC does everything else that I want. It even kills itself after a certain time, but in an unexpected way, which means I need to come up with a work-around. More on how to configure VLC and kill the scheduled stream in the next post.

•  Take Airtime Pro for a test spin. This month we are offering one month free with streaming. Just use the promo code wrd2014 for World Radio Day.