tags/linuxunfolding disastershttp://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/unfolding disastersikiwiki2011-12-07T18:58:39Zmailcaphttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/mailcap/2011-12-02T01:43:14Z2011-12-02T01:43:14Z
<p>Mailcap files, defined in <a href="http://tools.ietf.org/html/rfc1524">RFC 1524</a>, allow you to tell you
mail clients, web browsers, and other programs how you want to view
and edit various MIME types. Since interaction with your mailcap
files often occurs deep within the bowels of your program, I've
written up a very simple <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Python/">Python</a> script to test your mailcap
entries: <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/mailcap/mailcap-test.py">mailcap-test.py</a>. Enjoy!</p>
Postfixhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/Postfix/2011-11-16T00:48:33Z2011-11-16T00:48:33Z
<p>I spent some time today configuring <a href="http://www.postfix.org/">Postfix</a> so I could send mail
from home via <span class="createlink">SMTPS</span>. Verizon, our ISP, blocks port 25 to
external domains, forcing all outgoing mail through their
<code>outgoing.verizon.net</code> exchange server. In order to accept mail, they
also require you authenticate with your Verizon username and password,
so I wanted to use an encrypted connection.</p>
<p>For the purpose of this example, our Verizon username is <code>jdoe</code>, our
Verizon password is <code>YOURPASS</code>, you're running a local Postfix server
on <code>mail.example.com</code> for your site at <code>example.com</code>, and <code>12345</code> is a
free local port.</p>
<pre><code># cat /etc/postfix/main.cf
myhostname = mail.example.com
relayhost = [127.0.0.1]:12345
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
sender_canonical_maps = hash:/etc/postfix/sender_canonical
# cat /etc/postfix/saslpass
[127.0.0.1]:12345 jdoe@verizon.net:YOURPASS
# postmap /etc/postfix/saslpass
# cat /etc/postfix/sender_canonical
root@mail.example.com jdoe@example.com
root@example.com jdoe@example.com
root@localhost jdoe@example.com
jdoe@mail.example.com jdoe@example.com
jdoe@localhost jdoe@example.com
# postmap /etc/postfix/sender_canonical
# cat /etc/stunnel/stunnel.conf
[smtp-tls-wrapper]
accept = 12345
client = yes
connect = outgoing.verizon.net:465
# /etc/init.d/stunnel restart
# postfix reload
</code></pre>
<p>Test with:</p>
<pre><code>$ echo 'testing 1 2' | sendmail you@somewhere.com
</code></pre>
<p>Here's what's going on:</p>
<ul>
<li>You hand an outgoing message to your local Postfix, which decides to
send it via port <code>12345</code> on your localhost (<code>127.0.0.1</code>) (<code>relayhost</code>).</li>
<li>Stunnel picks up the connection from Postfix, encrypts everything,
and forwards the connection to port 465 on <code>outgoing.verizon.net</code>
(<code>stunnel.conf</code>).</li>
<li>Postfix identifies itself as <code>mail.example.com</code> (<code>myhostname</code>), and
authenticates using your Verizon credentials (<code>smtp_sasl_…</code>).</li>
<li>Because Verizon is picky about the <code>From</code> addresses it will accept,
we use <code>sender_canonical</code> to map addresses to something simple that
we've tested.</li>
</ul>
<p>And that's it :p. If you're curious, there's more detail about all
the Postfix config options in the <a href="http://www.postfix.org/postconf.5.html">postconf</a> man page, and there's
good SASL information in the <a href="http://www.postfix.org/SASL_README.html">SASL_README</a>.</p>
<p>There's also a <a href="http://www.zulius.com/how-to/set-up-postfix-with-a-remote-smtp-relay-host/">blog post by Tim White</a> which I found useful.
Because Verizon lacks <a href="http://en.wikipedia.org/wiki/STARTTLS">STARTTLS</a> support, his approach didn't work
for me out of the box.</p>
SMTPhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/SMTP/2011-11-16T00:45:35Z2011-11-16T00:45:35Z
<p>Verizon blocks outgoing connections on port 25 (<a href="http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a>) unless you
are connecting to their <code>outgoing.verizon.net</code> message exchange
server. This server requires authentication with your Verzon
username/password before it will accept your mail. For the purpose of
this example, our Verizon username is <code>jdoe</code>, our Verizon password is
<code>YOURPASS</code>, and were sending email from <code>me@example.com</code> to
<code>you@target.edu</code>.</p>
<pre><code>$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
mail from: <jdoe@example.com>
550 5.7.1 Authentication Required
quit
221 2.3.0 Bye received. Goodbye.
</code></pre>
<p>Because authenticating over an unencrypted connection is a Bad Idea™,
I was looking for an encrypted way to send my outgoing email.
Unfortunately, Verizon's exchange server does not support <a href="http://en.wikipedia.org/wiki/STARTTLS">STARTTLS</a>
for encrypting connections to <code>outgoing.verizon.net:25</code>:</p>
<pre><code>$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173003pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP E9B7EB199A9B52CF7D936A4DD3199D6F
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
</code></pre>
<p>Verizon <a href="http://www22.verizon.com/residentialhelp/fiosinternet/email/setup+and+use/questionsone/86782.htm">recommends</a> pre-STARTTLS approach of wrapping the
whole SMTP connection in TLS (<a href="http://en.wikipedia.org/wiki/SMTPS">SMTPS</a>), which it provides via
<code>outgoing.verizon.net:465</code>:</p>
<pre><code>$ python -c 'from base64 import *; print b64encode("\0jdoe@verizon.net\0YOURPASS")'
AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M=
$ openssl s_client -connect outgoing.verizon.net:465
...
220 vms173013pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173013pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP 9380A5843FE933CF9BD037667F4C950D
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
auth plain AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M
235 2.7.0 plain authentication successful.
mail from: <me@example.com>
250 2.5.0 Address Ok.
rcpt to: <you@target.edu>
250 2.1.5 you@target.edu OK.
data
354 Enter mail, end with a single ".".
From: Me <me@example.com>
To: You <you@target.edu>
Subject: testing
hello world
.
250 2.5.0 Ok, envelope id 4BHMFEZ7PHSETMT6@vms173013.mailsrvcs.net
quit
221 2.3.0 Bye received. Goodbye.
closed
</code></pre>
<p>This works, but with the rise of STARTTLS, getting your local
<a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Postfix/">Postfix</a> mail server to support SMTPS requires a bit of
<a href="http://www.postfix.org/TLS_README.html#client_smtps">fancyness</a> with <span class="createlink">stunnel</span>. The stunnel workaround is not too
complicated, but I also wanted to look into the <a href="http://tools.ietf.org/html/rfc4409">submission</a>
protocol (port 587), which adapts SMTP (designed for message transfer)
into a similar protocol for message submission. Unfortunately,
Verizon does not support STARTTLS here either.</p>
<pre><code>$ nc outgoing.verizon.net 587
220 vms173005.mailsrvcs.net -- Server ESMTP (...)
ehlo example.com
250-vms173005.mailsrvcs.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP DA941C5B31BE4B102BB69B809BC66C4A
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
</code></pre>
<p>In conclusion, Verizon supports a number of email submission
standards, but the only secure approach is to use the outdated SMTPS.
See my <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Postfix/">Postfix</a> post for details on configuring Postfix to use
Verizon's server for outgoing mail.</p>
<p>There are a number of good SMTP authentication tutorials out there. I
used <a href="http://qmail.jms1.net/test-auth.shtml">John Simpson</a> and <a href="http://www.fehcom.de/qmail/smtpauth.html">Erwin Hoffmann's</a> tutorials. For
cleaner examples of my testing tools (<code>nc</code> and <code>openssl s_client</code>),
see my <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Simple_servers/">simple servers</a> post.</p>
PDF bookmarks with Ghostscripthttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/PDF_bookmarks_with_Ghostscript/2011-10-01T12:40:50Z2011-09-26T17:57:47Z
<p>I was recently trying to add bookmarks to a <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/PDF/">PDF</a> I'd generated with
<a href="http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/">pdftk</a>. It turns out to be fairly simple to add bookmarks to a PDF
using <a href="http://ghostscript.com/">Ghostscript</a>, following <a href="http://ubuntuforums.org/showthread.php?t=1545064">maggoteer's post</a> to the
Ubunto forums. The syntax is:</p>
<pre><code>$ gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=out.pdf in-*.pdf pdfmarks
</code></pre>
<p>Where <code>out.pdf</code> is the generated PDF, <code>in-*.pdf</code> are the input PDFs,
and <code>pdfmarks</code> is a text file with contents like:</p>
<pre><code>[/Title (Title Page) /Page 1 /OUT pdfmark
[/Title (Table of Contents) /Page 3 /OUT pdfmark
...
</code></pre>
<p>Nice and easy.</p>
<p>For nested levels, use the <code>/Count</code> attribute. For example:</p>
<pre><code>[/Count 3 /Title (Chapter 1) /Page 1 /OUT pdfmark
[/Count -2 /Title (Section 1.1) /Page 2 /OUT pdfmark
[/Title (Section 1.1.1) /Page 3 /OUT pdfmark
[/Title (Section 1.1.2) /Page 4 /OUT pdfmark
[/Count -1 /Title (Section 1.2) /Page 5 /OUT pdfmark
[/Title (Section 1.2.1) /Page 6 /OUT pdfmark
[/Title (Section 1.3) /Page 7 /OUT pdfmark
</code></pre>
<p>The argument to <code>/Count</code> gives the number of immediately subordinate
bookmarks. The sign of the argument sets the default display
(negative for closed, positive for open).</p>
<p>You can also setup the document info dictionary with something like:</p>
<pre><code>[ /Title (My Test Document)
/Author (John Doe)
/Subject (pdfmark 3.0)
/Keywords (pdfmark, example, test)
/DOCINFO pdfmark
</code></pre>
<p>If you want more detail, take a look at <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/pdfmark_reference.pdf">Adobe's pdfmark
reference</a>.</p>
<p>I've bundled the whole pdfmarks-generation bit into a script,
<a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/PDF_bookmarks_with_Ghostscript/pdf-merge.py">pdf-merge.py</a>, which generates the pdfmark file and runs
Ghostscript automatically. Think of it as a bookmark-preserving
version of pdftk's <code>cat</code>. The script uses pdftk internally to extract
bookmark information from the source PDFs.</p>
<p>The script also adds a bit of PostScript to ignore any bookmarks in
the source PDFs during the Ghostscript run. The only bookmarks in the
output will be the ones you specify explicitly in the pdfmarks file.
If for some reason the automatically generated pdfmarks are not quite
what you want, the script can pause (via <code>--ask</code>) to allow you to
tweak the pdfmarks manually before running Ghostscript.</p>
Burning audio CDs with replay gainhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/Burning_audio_CDs_with_replay_gain/2011-09-23T05:37:18Z2011-09-23T05:37:18Z
<p>I've been streamlining my procedure for burning audio CDs, and I like
what I've come up with. Unfortunately, I'll never remember it on my
own, so here's a note to myself (any whoever else cares) on what to
do.</p>
<p>First, build your playlist in <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/MPD/">MPD</a>, using <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/MPD_playlist_duration/">my
one-liner</a> to calculate the playlist duration
(my CD-Rs hold 80 minutes). When you've got your playlist arranged to
your satisfaction, convert the files to <a href="http://en.wikipedia.org/wiki/WAV">WAV</a>s using:</p>
<pre><code>$ i=1; for x in $(mpc -f '%file%' playlist); do j=$(seq -f '%02g' $i $i); let "i += 1"; flac -d --apply-replaygain-which-is-not-lossless=t -o $j.$(basename ${x/.flac/.wav}) /var/lib/mpd/music/$x; done
</code></pre>
<p>This assumes that all the files in your playlist are <a href="http://flac.sourceforge.net/">FLAC</a> files,
which is a good idea (disk space is cheap, FLAC is lossless with good
open source support). It also assumes you've already stored
ReplayGain settings in your FLAC files. If you haven't, you'll get
warnings like:</p>
<pre><code>WARNING: can't get track (or even album) ReplayGain tags
</code></pre>
<p>If that happens, go back and read about <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Replay_gain/">replay gain</a>, add the tags,
and try again ;).</p>
<p>See my <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/Replay_gain/">replay gain</a> post for more details on the
<code>--apply-replaygain-which-is-not-lossless</code> option. After the decoding
step, you'll have a directory full of WAVs that have been normalized
to a standard track-level loudness. The bit about <code>i</code> and <code>j</code> ensures
that <code>*.wav</code> will list the tracks in the order in which they appear in
your playlist. Then burn the tracks to a CD using
<a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/cdrtools/">cdrecord</a>:</p>
<pre><code>$ cdrecord -v speed=1 dev=/dev/cdrom -eject -dao -audio -pad *.wav
</code></pre>
<p>If you don't care about the order, use <code>$(ls *.wav | shuf)</code> instead of
<code>*.wav</code>.</p>
<p>That's it! Audio CDs from MPD playlists in two lines.</p>
cookbookhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/cookbook/2011-08-05T17:59:01Z2011-08-05T17:59:01Z
<p><span class="infobox">
Available in a <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../git/">git</a> repository.<br />
Repository: <a href="http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cookbook.git" rel="vcs-git" title="cookbook repository">cookbook</a><br />
Author: W. Trevor King<br />
</span></p>
<p>I've been running a home-rolled recipe webapp for a year now, and it
worked fairly well in a bare-bones sort of way. However, I recently
had to make some changes to my personal website (since EveryDNS and
aparently most other free DNS providers were bought by Dyn), which
prompted me to translate <code>cookbook</code> into a <a href="https://www.djangoproject.com/">Django</a> app. Thanks to
the wonders of Django, <a href="https://github.com/sehmaschine/django-grappelli">Grappelli</a>, and <a href="https://github.com/alex/django-taggit">django-taggit</a>, the code
is now leaner, meaner, and prettier!</p>
<p>See the <a href="http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cookbook.git;a=blob;f=README;hb=HEAD">README</a> for details. </p>
h5confighttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/h5config/2011-07-28T16:25:06Z2011-07-28T16:25:06Z
<p><span class="infobox">
Available in a <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../git/">git</a> repository.<br />
Repository: <a href="http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=h5config.git" rel="vcs-git" title="h5config repository">h5config</a><br />
Author: W. Trevor King<br />
</span></p>
<p>Since the number of packages mooching off <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/pypiezo/">pypiezo</a>'s configuration
scheme was growing, I've split it out into it's own package. Now
there's a general package for all your <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../../posts/HDF5/">HDF5</a>-based configuration
needs.</p>
<p>The <code>README</code> is posted on the <a href="http://pypi.python.org/pypi/h5config/">PyPI page</a>.</p>
pypidhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/pypid/2011-07-27T16:06:19Z2011-07-27T16:06:19Z
<p><span class="infobox">
Available in a <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../git/">git</a> repository.<br />
Repository: <a href="http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=pypid.git" rel="vcs-git" title="pypid repository">pypid</a><br />
Author: W. Trevor King<br />
</span></p>
<p>I've just finished rewriting my <a href="http://en.wikipedia.org/wiki/PID_controller">PID</a> temperature control package in
pure-Python, and it's now clean enough to go up on PyPI. Features:</p>
<ul>
<li>Backend-agnostic architecture. I've written a first-order process
with dead time (FOPDT) test backend and a <a href="http://code.google.com/p/pymodbus/">pymodbus</a>-based backend
for our Melcor MTCA controller, but it should be easy to plug in
your own custom backend.</li>
<li>The general PID controller will automatically tune your backend
using any of a variety of tuning rules.</li>
</ul>
<p>The <code>README</code> is posted on the <a href="http://pypi.python.org/pypi/pypid/">PyPI page</a>.</p>
SciPy and ATLAShttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/SciPy_and_ATLAS/2011-12-07T18:58:39Z2011-07-25T10:55:26Z
<p>I was suprised the the other day by a <a href="http://www.scipy.org/">SciPy</a>-linking issue:</p>
<pre><code>$ python -c 'import scipy.linalg'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/scipy/linalg/__init__.py", line 9, in <module>
from basic import *
File "/usr/lib/python2.7/site-packages/scipy/linalg/basic.py", line 14, in <module>
from lapack import get_lapack_funcs
File "/usr/lib/python2.7/site-packages/scipy/linalg/lapack.py", line 15, in <module>
from scipy.linalg import clapack
ImportError: /usr/lib/python2.7/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv
</code></pre>
<p>Searching around, it turns out there have been <a href="https://bugs.gentoo.org/show_bug.cgi?id=129524">problems like
this</a> for a while, and it's flared up <a href="http://archives.gentoo.org/gentoo-science/msg_83c0a72f758099201944bdccd401603a.xml">again</a>
recently. There's a <a href="https://bugs.gentoo.org/show_bug.cgi?id=371099">new Gentoo bug</a> tracking the most recent
issues. In general, <a href="http://numpy.scipy.org/">NumPy</a>/<a href="http://www.scipy.org/">SciPy</a> seems to be picky about the
particular <a href="http://www.netlib.org/blas/">BLAS</a> implementation you're using, and you'll get
problems like the above if you're using the reference implementations
of BLAS and <a href="http://www.netlib.org/lapack/">LAPACK</a>.</p>
<pre><code>$ for x in blas cblas lapack; do eselect $x list; done
Installed BLAS for library directory lib
[1] reference *
Installed CBLAS for library directory lib
[1] gsl
[2] reference *
Installed LAPACK for library directory lib
[1] reference *
</code></pre>
<p>You can fix the problem by installing and selecting the <a href="http://math-atlas.sourceforge.net/">ATLAS</a>
libraries</p>
<pre><code>$ sudo emerge -av blas-atlas lapack-atlas
$ for x in blas cblas lapack; do sudo eselect $x set atlas; done
</code></pre>
<p>after which you should re-install any packages that may need to be relinked.</p>
<pre><code>$ sudo emerge -av numpy scipy
</code></pre>
<p>Now you can import the linear algebra module without crashing:</p>
<pre><code>$ python -c 'import scipy.linalg'
</code></pre>
insiderhttp://www.physics.drexel.edu/~wking/unfolding-disasters/posts/insider/2011-07-21T22:35:46Z2011-07-21T22:30:33Z
<p><span class="infobox">
Available in a <a href="http://www.physics.drexel.edu/~wking/unfolding-disasters/tags/linux/../git/">git</a> repository.<br />
Repository: <a href="http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=insider.git" rel="vcs-git" title="insider repository">insider</a><br />
Author: W. Trevor King<br />
</span></p>
<p>Insider is a little <a href="https://www.djangoproject.com/">Django</a> app I wrote to help my brother,
Garrett, track insider trading with a simple, familiar web interface.
It's a pretty simple app, partly thanks to Bradley Ayers'
<a href="https://github.com/bradleyayers/django-tables2/">django-tables2</a>, which does the table formatting. Just goes to
show that a good scripting language and framework make developing
simple apps a breeze!</p>
<p>The <code>README</code> is posted on the <a href="http://pypi.python.org/pypi/insider/">PyPI page</a>.</p>