Create a DivX Plus HD video file on Linux

For years, I have been recording shows and films on TV. For this, I have been using my old “Pinnacle PCTV Stereo” acquisition card, inserted into a 1GHz VIA Nehemiah processor-powered living-room PC. More recently, I have kept using this card from time to time in a 1.8GHz Core2Duo E4300-powered desktop PC.

The living-room PC is no more; its power supply overheated. Now, I am using a regular living-room media player, which is able to read my media through the network using the DLNA protocol. This hardware media player does not have the same tolerance for varying file formats as seen with a standard PC, but luckily it can read the “DivX Plus HD” file format, which is perfect as far as I am concerned: based on a Matroska (.mkv) container, it can include chapters, and subtitles, etc.

Cet article existe aussi en français.

Since my video acquisition card is of the low-spec. kind, and my former living-room PC was also low-powered, my recordings were always made in an almost-raw (and space-greedy) format: an MJPEG video stream, and a raw audio stream, inside an AVI container. Then a two-pass transcode step would convert such files into more compact files. I mostly used 3  final formats:

  • a few films were in SVCD format, usually intended to be burnt to CD,
  • a few films were in DVD format (in particular old VHS tapes that I digitalized), usually intended to be burnt to DVD,
  • and most recordings ended up in the MPEG2 format, which is less strict and more space-efficient, even though I knew these would only be readable with a computer.

Of course, this third format is the one that cannot be read by my shiny new DLNA media player, whence the need to transcode these files. And since the Matroska container allows the embedding of subtitles, I can even convert my family video files, for which “subtitles” are really my notes about who is on the screen, and where ;-)

After several tries, it became apparent that creating a file that would be compatible with a hardware —not software— media player is tricky. On the one hand, the DivX Plus HD standard is not always followed to the letter; on the other hand, some aspects of this standard, that may seem insignificant details, are mandatory for a working result.

I finally found a combination of tools and parameters, with which I achieve a fully compatible result (with fast-forward, fast-rewind, et al.) with 100% success. Here is the script that I wrote while working towards this goal:

#!/usr/bin/bash
# Needed:
# ffmpeg, ffprobe, mkvmerge, x264, bash, bc, awk, sed, sort, uniq, tr
# Mandatory:
# -i <file> [-i <file> …] Input files
# Optional:
# -h Print this help, and information about the inputs.
# -n Do not actually run the commands (print them).
# -y Accept the default answers to this program’s
# questions: include all streams, use the first
# default audio in the inputs as default audio in the
# output (same for subtitles), and mark any audio or
# subtitles stream that is forced in the inputs as
# forced in the output.
# -o <file> Output file (default: output.mkv).
# -t <temporary storage> Place where intermediate files will be stored.
# -A <kb/s> kb/s for each audio channel (default: 64).
# -V <kb/s> | -Q <quality> kb/s for the video (default: 850, with 2 passes),
# or constant quality between 0 and 51
# (see: http://slhck.info/articles/crf).
# -S Slow first pass (“turbo”-1st pass sometimes fails).
# -P <speed preset> Preset, among “ultrafast”, “superfast”,
# “veryfast”, “faster”, “fast”, “medium”,
# “slow”, “slower”, “veryslow”, “placebo”
# (the slower the better; default: medium).
# -T <type tuning> Kind of video, among “film”, “animation”,
# “grain”, “stillimage”, “psnr”, “ssim”,
# “fastdecode”, “zerolatency”.
# -W <max width> Maximum width (greater than 1920 is useless).
# -H <max height> Maximum height (greater than 1080 is useless).
# -C Crop black borders using ffmpeg’s autodetection.

inputs=()
output=output.mkv
tmpdir=/tmp
x264slow=
x264qual="--bitrate 850 --pass"
x264maxW=1920
x264maxH=1080
abort=
fake=
defaults=
autocrop=
error=0

## READ PARAMETERS

function usage() {
sed -n "2,/^\$/s/.//p" "$0"
abort=true
}

while getopts i:o:t:A:SV:Q:P:T:W:H:Chny opt; do case $opt in
h) usage ;;
n) fake=true ;;
y) defaults=true ;;
i) inputs[${#inputs[*]}]="$OPTARG" ;;
o) output="$OPTARG" ;;
t) if [ -d "$OPTARG" ] && [ -w "$OPTARG" ] \
&& [ "$OPTARG" == "$(printf '%q' "$OPTARG")" ]; then
tmpdir="${OPTARG%/}"
else
echo "$OPTARG contains special characters, or cannot be written." >&2
fi ;;
A) akbps="${OPTARG//[^0-9]}" ;;
S) x264slow=true ;;
V) OPTARG="${OPTARG//[^0-9]}"; x264qual="--bitrate ${OPTARG:-850} --pass" ;;
Q) OPTARG="${OPTARG//[^0-9]}"
[ -n "$OPTARG" ] && x264qual="--crf $OPTARG" \
|| x264qual="--bitrate 850 --pass" ;;
P) preset="$OPTARG" ;;
T) tune="$OPTARG" ;;
W) OPTARG="${OPTARG//[^0-9]}"; x264maxW=${OPTARG:-1920}
[ $x264maxW -le 1920 ] || x264maxW=1920
[ $x264maxW -ge 320 ] || x264maxW=320 ;;
H) OPTARG="${OPTARG//[^0-9]}"; x264maxH=${OPTARG:-1080}
[ $x264maxH -le 1080 ] || x264maxH=1080
[ $x264maxH -ge 240 ] || x264maxH=240 ;;
C) autocrop=true ;;
esac; done
if [ -z "$abort" ] && [ ${#inputs[*]} -eq 0 ]; then usage >&2; exit 1; fi

tmppfx=$tmpdir/divx+hd.$$
if [[ "$x264qual" =~ pass ]]; then
x264qual="--stats $tmppfx.stats $x264qual"
fi

## ANALYSE INPUTS

# fields: str:file path, bool:use chapters, bool:use some subtitles,
# int:number of chapters
IFS=$'\n' read -d '' -a f_info < <(
for f in "${inputs[@]}"; do
printf "$f\t0\t0\t"
ffprobe -loglevel error -show_chapters "$f" | grep -Fx '[CHAPTER]' | wc -l
done
)

# fields: int:file index, int:stream index, str:codec, str:profile, int:width,
# int:height, [str:transformation], [int:new width], [int:new height],
# str:sample aspect ratio, str:fps, float:stream start time,
# str:language, [str:title], bool:use this stream
exec 3>&1
IFS=$'\n' read -d '' -a v_info < <(
for ((i=0; i<${#inputs[*]}; i++)); do
ffprobe -loglevel error -show_streams -select_streams v "${inputs[$i]}" \
| awk -F= -vfile=$i -vmw=$x264maxW -vmh=$x264maxH \
-vffcrop=$autocrop -vfffile="${inputs[$i]}" -vOFS=$'\t' '
BEGIN{
gsub(/["\$]/,"\\\\&",fffile)
}
/^\[STREAM/ {
idx=0; codec=""; profile=""; fullw=768; fullh=432
sar="1:1"; fps=25; start=0; lang="und"; title=""
}
$1=="index" { idx=$2 }
$1=="codec_name" { codec=$2 }
$1=="profile" { profile=$2 }
$1=="width" { fullw=$2 }
$1=="height" { fullh=$2 }
$1=="sample_aspect_ratio" { sar=$2 }
$1=="avg_frame_rate" { fps=$2 }
$1=="start_time" { start=$2 }
$1=="TAG:language" { lang=$2 }
$1=="TAG:title" { title=$2 }
/^\[\/STREAM/ {
w=fullw; h=fullh; stdop=""; stdw=0; stdh=0
cropL=0; cropT=0; cropR=0; cropB=0
if (ffcrop!="") {
fmt="ffmpeg -i \"%s\" -map 0:%d -t 00:30:00 -vf crop="
fmt=fmt (w-4) ":" (h-4) ":2:2,cropdetect=0.1:2:1 -f null /dev/null 2>&1"
fmt=fmt " | sed -n \"s/.*crop=//p\" | sort | uniq -c | sort -k1,1n"
fmt=fmt " | awk \"END{print \\$2}\" | tr : ="
cmd=sprintf(fmt, fffile, idx)
print cmd >"/dev/fd/3"
cmd | getline; close(cmd)
if ($1+4!=w || $2+4!=h) {
stdop="crop⇒/"
w=$1; h=$2; stdw=w; stdh=h
cropL=2+2*($3/2); cropT=2+2*($4/2)
cropR=fullw-w-cropL; cropB=fullh-h-cropT
}
}
if (w>mw || h>mh) {
stdop=stdop "resize⇒/"
if ((h*mw/w)<mh) {
stdw=mw; stdh=int(h*mw/(w*8)+0.5)*8; if (stdh<240) stdh=240
} else {
stdw=int(w*mh/(h*8)+0.5)*8; stdh=mh; if (stdw<320) stdw=320
}
} else if (w<320 || h<240) {
stdop=stdop "resize⇒/"
if ((h*320/w)>240) {
stdw=320; stdh=int(h*320/(w*8)+0.5)*8; if (stdh>mh) stdh=mh
} else {
stdw=int(w*240/(h*8)+0.5)*8; stdh=240; if (stdw>mw) stdw=mw
}
} else if (w/8!=int(w/8) || h/8!=int(h/8)) {
stdop="crop⇒/"
stdw=int(w/8)*8; stdh=int(h/8)*8
halfcropw=2*int((w-stdw)/4); halfcroph=2*int((h-stdh)/4)
cropL=cropL+halfcropw; cropR=cropR+(w-stdw-halfcropw)
cropT=cropT+halfcroph; cropB=cropB+(h-stdh-halfcroph)
}
print file, idx, codec, profile, fullw, fullh, stdop, cropL, cropT, \
cropR, cropB, stdw, stdh, sar, fps, start, lang, title, 0
}'
done
)
exec 3>&-

# fields: int:file index, int:stream index, str:codec, str:profile,
# int:sample frequency, int:number of channels, int:planned kb/s,
# str:layout, float:stream start time, bool:stream is default in file,
# bool:stream is forced in file, bool:stream is for visual impaired,
# str:language, [str:title], bool:use this stream,
# bool:use this stream as default audio, bool:force this audio stream
IFS=$'\n' read -d '' -a a_info < <(
for ((i=0; i<${#inputs[*]}; i++)); do
ffprobe -loglevel error -show_streams -select_streams a "${inputs[$i]}" \
| awk -F= -vfile=$i -vabr=${akbps:-64} -vOFS=$'\t' '
/^\[STREAM/ {
idx=0; codec=""; profile=""; sampfreq=0; channels=2; layout="2.0"
start=0; def=0; force=0; seehelp=0; lang="fre"; title=""
}
$1=="index" { idx=$2 }
$1=="codec_name" { codec=$2 }
$1=="profile" { profile=$2 }
$1=="sample_rate" { sampfreq=$2 }
$1=="channels" { channels=$2 }
$1=="channel_layout" { layout=$2 }
$1=="start_time" { start=$2 }
$1=="DISPOSITION:default" { def=$2 }
$1=="DISPOSITION:forced" { force=$2 }
$1=="DISPOSITION:visual_impaired" { seehelp=$2 }
$1=="TAG:language" { lang=$2 }
$1=="TAG:title" { title=$2 }
/^\[\/STREAM/ {
print file, idx, codec, profile, sampfreq, channels, abr*channels, \
layout, start, def, force, seehelp, lang, title, 0, 0, 0
}'
done
)

# fields: int:file index, int:stream index, str:codec, float:stream start time,
# bool:stream is default in file, bool:stream is forced in file,
# bool:stream is for hearing impaired, str:language, [str:title],
# bool:use these subtitles, bool:use this stream as default subtitles,
# bool:force these subtitles
IFS=$'\n' read -d '' -a s_info < <(
for ((i=0; i<${#inputs[*]}; i++)); do
ffprobe -loglevel error -show_streams -select_streams s "${inputs[$i]}" \
| awk -F= -vfile=$i -vOFS=$'\t' '
/^\[STREAM/ {
idx=0; codec=""; start=0; def=0; force=0; hearhelp=0; lang="und"; title=""
}
$1=="index" { idx=$2 }
$1=="codec_name" { codec=$2 }
$1=="start_time" { start=$2 }
$1=="DISPOSITION:default" { def=$2 }
$1=="DISPOSITION:forced" { force=$2 }
$1=="DISPOSITION:hearing_impaired" { hearhelp=$2 }
$1=="TAG:language" { lang=$2 }
$1=="TAG:title" { title=$2 }
/^\[\/STREAM/ {
print file, idx, codec, start, def, force, hearhelp, lang, title, 0, 0, 0
}'
done
)

## PRINT INFORMATION, SELECT STREAMS

# $1:array, $2:list-program (awk)
function print_info() {
local -n arr=$1
printf '%s\n' "${arr[@]}" | awk -F$'\t' -vOFS=$'\t' "$2"
}

# $1:array, $2:question, $3:default answer, $4:field to set,
# $5==1 if only one flag can be set,
# $6>0 means that the flag with this number must be set in the parent file, too.
function set_flags() {
local -n arr=$1
local choice="$3"
local c=
local f=
if [ -z "$defaults" ]; then
read -e -i "$choice" -p "$2" choice
else
echo "$2$choice"
fi
for c in $choice; do
c=${c//[^0-9]}
if [ -n "$c" ] && [ $c -ge 0 ] && [ $c -lt ${#arr[*]} ]; then
arr[$c]="$(awk -F$'\t' -vOFS=$'\t' -vf=$4 '{$f=1;print}' <<<"${arr[$c]}")"
if [ -n "$6" ]; then
f=$(awk -F $'\t' '{print $1}' <<<"${s_info[$c]}")
f_info[$f]="$(
awk -F$'\t' -vOFS=$'\t' -vf=$6 '{$f=1;print}' <<<"${f_info[$f]}")"
fi
fi
[ "$5" == "1" ] && break
done
}

if [ ${#f_info[*]} -gt 0 ]; then
echo FILES:
print_info f_info '{ print NR-1, $1, $4==0?"":("(with " $4 " chapters)") }'
if [ -z "$abort" ]; then
set_flags f_info 'File from which chapters should be read (if any): ' \
"$(printf '%s\n' "${f_info[@]}" \
| awk -F$'\t' '$4>0{print NR-1; exit}')" 2 1
fi
fi
if [ ${#v_info[*]} -gt 0 ]; then
echo VIDEO:
print_info v_info '
BEGIN {print "scale=2" |& "bc -q"}
{ sub(/.$/,"",$7)
print $15 |& "bc -q"; "bc -q" |& getline fps
print NR-1, "file " $1, $3 ($4==""?"":(" (" $4 ")")), $5 "×" $6, \
$7==""?"":($7 $12 "×" $13), fps "fps", $17, $18 }'
if [ -z "$abort" ]; then
set_flags v_info 'Space-separated list of video streams to include: ' \
"$(eval echo \{0..$((${#v_info[*]}-1))\})" 19
fi
fi
if [ ${#a_info[*]} -gt 0 ]; then
echo AUDIO:
print_info a_info '
{ print NR-1, "file " $1, $3 ($4==""?"":(" (" $4 ")")), \
$6 "×@" $5 "Hz", $8, $7 "kb/s", $13, $14, \
$10=="1"?"(default track)":"", $11=="1"?"(forced track)":"", \
$12=="1"?"(vision impaired)":"" }'
if [ -z "$abort" ]; then
set_flags a_info 'Space-separated list of audio streams to include: ' \
"$(eval echo \{0..$((${#a_info[*]}-1))\})" 15
set_flags a_info 'Default audio stream (if any): ' \
"$(printf '%s\n' "${a_info[@]}" \
| awk -F$'\t' '$10==1&&$15==1{print NR-1}' | head -n 1)" 16 1
set_flags a_info 'Forced audio streams (if any): ' \
"$(printf '%s\n' "${a_info[@]}" \
| awk -F$'\t' -vORS=' ' '$11==1&&$15==1{print NR-1}')" 17
fi
fi
if [ ${#s_info[*]} -gt 0 ]; then
echo SUBTITLES:
print_info s_info '
{ print NR-1, "file " $1, $3, $8, $9, \
$5=="1"?"(default track)":"", $6=="1"?"(forced track)":"", \
$7=="1"?"(hearing impaired)":"" }'
if [ -z "$abort" ]; then
set_flags s_info 'Space-separated list of subtitle streams to include: ' \
"$(eval echo \{0..$((${#s_info[*]}-1))\})" 10 0 3
set_flags s_info 'Default subtitle stream (if any): ' \
"$(printf '%s\n' "${s_info[@]}" \
| awk -F$'\t' '$5==1&&$10==1{print NR-1}' | head -n 1)" 11 1
set_flags s_info 'Forced subtitle streams (if any): ' \
"$(printf '%s\n' "${s_info[@]}" \
| awk -F$'\t' -vORS=' ' '$6==1&&$10==1{print NR-1}')" 12
fi
fi
[ -z "$abort" ] || exit 0

## ENCODE

function split() {
# needed because `read` merges adjacent field separators
fields="$1"; shift
while IFS=$'\n' read field; do
[ $# -gt 0 ] && eval "${1}=\"$(sed 's/[\"$]/\\\0/g' <<<"$field")\""; shift
done < <(tr '\t' '\n' <<<"$fields")
}
function run() {
local err=0
for item in "$@"; do printf ' %q' "$item"; done; printf '\n'
if [ -z "$fake" ]; then
"$@"
err=$?
if [ $err -gt 1 ] || [ $err -gt 0 -a "$1" != mkvmerge ]; then
echo "ERROR: $1 exited with code ${err}." >&2
error=$((error+1))
fi
fi
}
function on_exit() {
if [ $error -eq 0 ]; then
run rm -f ${tmppfx}*
else
echo 'Errors were encountered. These files were not deleted:' >&2
printf '%s\n' ${tmppfx}* >&2
fi
}
trap on_exit EXIT

#⇒ Video
#⇒ http://labs.divx.com/node/16598

for v in "${v_info[@]}"; do
split "$v" \
file idx codec profile w h stdop cropL cropT cropR cropB stdw stdh \
sar fps start lang title use
[ $use -eq 1 ] || continue
if [ $(bc <<<"$fps") -gt 25 ]; then
fps=25
fi
vopt_divx="--8x8dct --vbv-maxrate=20000 --vbv-bufsize=25000 --level 40 --bframes 3 --keyint $(bc <<<"4*$fps")"
vopt="${preset:+--preset $preset }${tune:+--tune $tune }--fps $fps --sar $sar"
[[ "$stdop" =~ crop ]] && stdop=${stdop/⇒/:$cropL,$cropT,$cropR,$cropB}
[[ "$stdop" =~ resize ]] && stdop=${stdop/⇒/:$stdw,$stdh}
[ -n "$stdop" ] && vopt="$vopt --vf ${stdop:0:-1}"

if [[ "$x264qual" =~ pass ]]; then
run nice -n 10 x264 $vopt $vopt_divx $x264qual 1 \
${x264slow:+--slow-firstpass} -o $tmppfx.$file.$idx.mkv "${inputs[$file]}"
run nice -n 10 x264 $vopt $vopt_divx $x264qual 2 \
-o $tmppfx.$file.$idx.mkv "${inputs[$file]}"
else
run nice -n 10 x264 $vopt $vopt_divx $x264qual \
-o $tmppfx.$file.$idx.mkv "${inputs[$file]}"
fi
done

#⇒ Audio
#⇒ https://trac.ffmpeg.org/wiki/Encode/AAC

for a in "${a_info[@]}"; do
split "$a" \
file idx codec profile sampfreq channels kbps layout start def force \
seehelp lang title use setdef setforce
[ $use -eq 1 ] || continue
if [ "$codec" == 'aac' ]; then
run ffmpeg -loglevel warning -i "${inputs[$file]}" -vn -map 0:$idx \
-movflags +faststart -c:a copy $tmppfx.$file.$idx.aac
else
run ffmpeg -loglevel warning -i "${inputs[$file]}" -vn -map 0:$idx \
-movflags +faststart -c:a libfdk_aac -b:a ${kbps}k $tmppfx.$file.$idx.aac
fi
done

## MUX DIVX PLUS HD

#⇒ http://www.videoredo.net/msgBoard/archive/index.php/t-31105.html
#⇒ https://github.com/mbunkus/mkvtoolnix/wiki/Playback-does-not-work-VLC-cannot-seek-mkvmerge-v5.9.0

mkvopt=(
--engage no_cue_duration,no_cue_relative_position
--clusters-in-meta-seek -o "$output" )

# subtitles & chapters

for f in "${f_info[@]}"; do
split "$f" path usechap usesub nbchap
[ $usechap -eq 0 -a $usesub -eq 0 ] && continue
if [ $usesub -eq 1 ]; then
subs=
newopt=()
for s in "${s_info[@]}"; do
split "$s" \
file idx codec start def force hearhelp lang title use setdef setforce
[ $use -eq 1 ] || continue
subs=$subs,$idx
newopt=( "${newopt[@]}"
--default-track $idx:$setdef --forced-track $idx:$setforce )
done
newopt=( -A -D -s ${subs:1} -B -M --no-global-tags "${newopt[@]}" )
else
newopt=( -A -D -S -T -B -M --no-global-tags )
fi
if [ $usechap -eq 0 ]; then
newopt[${#newopt[*]}]=--no-chapters
fi
mkvopt=( "${mkvopt[@]}" "${newopt[@]}" "$path" )
done

# video

for v in "${v_info[@]}"; do
split "$v" \
file idx codec profile w h stdop cropL cropT cropR cropB stdw stdh \
sar fps start lang title use
[ $use -eq 1 ] || continue
newopt=( -A -d 0 -S -B -M --no-chapters --no-global-tags \
--default-track 0:0 --forced-track 0:0 \
--track-name 0:"${title:-$(sed 's#.*/##;s#\.[^.]*##' <<<"$output") ($lang)}" \
--language 0:$lang $tmppfx.$file.$idx.mkv )
mkvopt=( "${mkvopt[@]}" "${newopt[@]}" )
done

# audio

for a in "${a_info[@]}"; do
split "$a" \
file idx codec profile sampfreq channels kbps layout start def force \
seehelp lang title use setdef setforce
[ $use -eq 1 ] || continue
newopt=( -a 0 -D -S -B -M --no-chapters --no-global-tags --default-track
0:$setdef --forced-track 0:$setforce --track-name
0:"${title:-$lang $layout}" --language 0:$lang )
if [ "$profile" == "HE-AAC" ]; then
newopt[${#newopt[*]}]=--aac-is-sbr
newopt[${#newopt[*]}]=0:1
fi
mkvopt=( "${mkvopt[@]}" "${newopt[@]}" $tmppfx.$file.$idx.aac )
done

# go!

run mkvmerge "${mkvopt[@]}"

This script’s behaviour is interactive, unless changed with parameters: questions are asked about which streams to include in the generated file. The default answers should be fine, and simply accepting these should be enough to get the wanted result. If that is indeed the case with your video files, then you can automate the script by giving to it the -y parameter, the role of which is to accept all default answers.

Note that this script makes some assumptions:

  • First, the video streams given for input are supposed to not already be in the x264 format with all required properties for DivX Plus HD; thus they will always be transcoded.
  • Next, the script is written to limit the frame-rate to 25 images per second (PAL), firstly because less frames means less bytes, hence more bytes available to improve overall quality; secondly because with more than 30 images per second, the standard defines different maximum dimensions for the frames, and I did not want to manage two different sub-standards.
  • Finally, ffmpeg is supposed to support the fdk_aac codec; if this is not the case, then “-c:a libfdk_aac” should be replaced with “-c:a aac” in the script.

This script is also intended to be kind-of educative. Not only can it run as an automatic transcoding tool, but it can also assist in manual transcoding.

This script’s default behaviour is to display each command before it is run. You can choose to only do the displaying part, and not run anything (-n parameter), so that you can see what would have been run but ensure that nothing is modified on the hard disk. You can then adapt the displayed commands and run them yourself.

You can ask for information as well. The -h parameter will tell you what information can be read from the input streams (-i parameter): video, audio, and subtitle streams, and chapters.

As you can see in the script itself, the web sites that were most helpful in reaching my goal were:

Changelog:

  • 2015-04-04 — I discovered by chance that the crop filter in x264 will only accept even values; for example, if I must remove 6 pixels, I have to remove 2 pixels on one side and 4 on the other side, instead of removing 3 pixels on each side. Script fixed.

Ajouter un commentaire

Le code HTML est affiché comme du texte et les adresses web sont automatiquement transformées.

La discussion continue ailleurs

URL de rétrolien : http://yalis.fr/cms/index.php/trackback/71

Fil des commentaires de ce billet