Linux-related pages, mostly about system administration.

Mailcap files, defined in RFC 1524, allow you to tell you mail clients, web browsers, and other programs how you want to view and edit various MIME types. Since interaction with your mailcap files often occurs deep within the bowels of your program, I've written up a very simple Python script to test your mailcap entries: mailcap-test.py. Enjoy!

Posted Thu Dec 1 20:43:14 2011 Tags: linux

I spent some time today configuring Postfix so I could send mail from home via SMTPS. Verizon, our ISP, blocks port 25 to external domains, forcing all outgoing mail through their outgoing.verizon.net exchange server. In order to accept mail, they also require you authenticate with your Verizon username and password, so I wanted to use an encrypted connection.

For the purpose of this example, our Verizon username is jdoe, our Verizon password is YOURPASS, you're running a local Postfix server on mail.example.com for your site at example.com, and 12345 is a free local port.

# cat /etc/postfix/main.cf
myhostname = mail.example.com
relayhost = [127.0.0.1]:12345
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
sender_canonical_maps = hash:/etc/postfix/sender_canonical
# cat /etc/postfix/saslpass
[127.0.0.1]:12345 jdoe@verizon.net:YOURPASS
    # postmap /etc/postfix/saslpass
    # cat /etc/postfix/sender_canonical
    root@mail.example.com jdoe@example.com
root@example.com jdoe@example.com
root@localhost jdoe@example.com
jdoe@mail.example.com jdoe@example.com
jdoe@localhost jdoe@example.com
    # postmap /etc/postfix/sender_canonical
# cat /etc/stunnel/stunnel.conf
[smtp-tls-wrapper]
accept = 12345
client = yes
connect = outgoing.verizon.net:465
# /etc/init.d/stunnel restart
# postfix reload

Test with:

$ echo 'testing 1 2' | sendmail you@somewhere.com

Here's what's going on:

  • You hand an outgoing message to your local Postfix, which decides to send it via port 12345 on your localhost (127.0.0.1) (relayhost).
  • Stunnel picks up the connection from Postfix, encrypts everything, and forwards the connection to port 465 on outgoing.verizon.net (stunnel.conf).
  • Postfix identifies itself as mail.example.com (myhostname), and authenticates using your Verizon credentials (smtp_sasl_…).
  • Because Verizon is picky about the From addresses it will accept, we use sender_canonical to map addresses to something simple that we've tested.

And that's it :p. If you're curious, there's more detail about all the Postfix config options in the postconf man page, and there's good SASL information in the SASL_README.

There's also a blog post by Tim White which I found useful. Because Verizon lacks STARTTLS support, his approach didn't work for me out of the box.

Posted Tue Nov 15 19:48:33 2011 Tags: linux

Verizon blocks outgoing connections on port 25 (SMTP) unless you are connecting to their outgoing.verizon.net message exchange server. This server requires authentication with your Verzon username/password before it will accept your mail. For the purpose of this example, our Verizon username is jdoe, our Verizon password is YOURPASS, and were sending email from me@example.com to you@target.edu.

$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
mail from: <jdoe@example.com>  
550 5.7.1 Authentication Required
quit
221 2.3.0 Bye received. Goodbye.

Because authenticating over an unencrypted connection is a Bad Idea™, I was looking for an encrypted way to send my outgoing email. Unfortunately, Verizon's exchange server does not support STARTTLS for encrypting connections to outgoing.verizon.net:25:

$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173003pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP E9B7EB199A9B52CF7D936A4DD3199D6F
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.

Verizon recommends pre-STARTTLS approach of wrapping the whole SMTP connection in TLS (SMTPS), which it provides via outgoing.verizon.net:465:

$ python -c 'from base64 import *; print b64encode("\0jdoe@verizon.net\0YOURPASS")'
AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M=
$ openssl s_client -connect outgoing.verizon.net:465
...
220 vms173013pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173013pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP 9380A5843FE933CF9BD037667F4C950D
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
auth plain AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M
235 2.7.0 plain authentication successful.
mail from: <me@example.com>
250 2.5.0 Address Ok.
rcpt to: <you@target.edu>
250 2.1.5 you@target.edu OK.
data
354 Enter mail, end with a single ".".
From: Me <me@example.com>
To: You <you@target.edu>
Subject: testing

hello world 
.
250 2.5.0 Ok, envelope id 4BHMFEZ7PHSETMT6@vms173013.mailsrvcs.net
quit
221 2.3.0 Bye received. Goodbye.
closed

This works, but with the rise of STARTTLS, getting your local Postfix mail server to support SMTPS requires a bit of fancyness with stunnel. The stunnel workaround is not too complicated, but I also wanted to look into the submission protocol (port 587), which adapts SMTP (designed for message transfer) into a similar protocol for message submission. Unfortunately, Verizon does not support STARTTLS here either.

$ nc outgoing.verizon.net 587
220 vms173005.mailsrvcs.net -- Server ESMTP (...)
ehlo example.com
250-vms173005.mailsrvcs.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP DA941C5B31BE4B102BB69B809BC66C4A
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.

In conclusion, Verizon supports a number of email submission standards, but the only secure approach is to use the outdated SMTPS. See my Postfix post for details on configuring Postfix to use Verizon's server for outgoing mail.

There are a number of good SMTP authentication tutorials out there. I used John Simpson and Erwin Hoffmann's tutorials. For cleaner examples of my testing tools (nc and openssl s_client), see my simple servers post.

Posted Tue Nov 15 19:45:35 2011 Tags: linux

I was recently trying to add bookmarks to a PDF I'd generated with pdftk. It turns out to be fairly simple to add bookmarks to a PDF using Ghostscript, following maggoteer's post to the Ubunto forums. The syntax is:

$ gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=out.pdf in-*.pdf pdfmarks

Where out.pdf is the generated PDF, in-*.pdf are the input PDFs, and pdfmarks is a text file with contents like:

[/Title (Title Page) /Page 1 /OUT pdfmark
[/Title (Table of Contents) /Page 3 /OUT pdfmark
...

Nice and easy.

For nested levels, use the /Count attribute. For example:

[/Count 3 /Title (Chapter 1) /Page 1 /OUT pdfmark
[/Count -2 /Title (Section 1.1) /Page 2 /OUT pdfmark
[/Title (Section 1.1.1) /Page 3 /OUT pdfmark
[/Title (Section 1.1.2) /Page 4 /OUT pdfmark
[/Count -1 /Title (Section 1.2) /Page 5 /OUT pdfmark
[/Title (Section 1.2.1) /Page 6 /OUT pdfmark
[/Title (Section 1.3) /Page 7 /OUT pdfmark

The argument to /Count gives the number of immediately subordinate bookmarks. The sign of the argument sets the default display (negative for closed, positive for open).

You can also setup the document info dictionary with something like:

[ /Title (My Test Document)
  /Author (John Doe)
  /Subject (pdfmark 3.0)
  /Keywords (pdfmark, example, test)
  /DOCINFO pdfmark

If you want more detail, take a look at Adobe's pdfmark reference.

I've bundled the whole pdfmarks-generation bit into a script, pdf-merge.py, which generates the pdfmark file and runs Ghostscript automatically. Think of it as a bookmark-preserving version of pdftk's cat. The script uses pdftk internally to extract bookmark information from the source PDFs.

The script also adds a bit of PostScript to ignore any bookmarks in the source PDFs during the Ghostscript run. The only bookmarks in the output will be the ones you specify explicitly in the pdfmarks file. If for some reason the automatically generated pdfmarks are not quite what you want, the script can pause (via --ask) to allow you to tweak the pdfmarks manually before running Ghostscript.

Posted Mon Sep 26 13:57:47 2011 Tags: linux

I've been streamlining my procedure for burning audio CDs, and I like what I've come up with. Unfortunately, I'll never remember it on my own, so here's a note to myself (any whoever else cares) on what to do.

First, build your playlist in MPD, using my one-liner to calculate the playlist duration (my CD-Rs hold 80 minutes). When you've got your playlist arranged to your satisfaction, convert the files to WAVs using:

$ i=1; for x in $(mpc -f '%file%' playlist); do j=$(seq -f '%02g' $i $i); let "i += 1"; flac -d --apply-replaygain-which-is-not-lossless=t -o $j.$(basename ${x/.flac/.wav}) /var/lib/mpd/music/$x; done

This assumes that all the files in your playlist are FLAC files, which is a good idea (disk space is cheap, FLAC is lossless with good open source support). It also assumes you've already stored ReplayGain settings in your FLAC files. If you haven't, you'll get warnings like:

WARNING: can't get track (or even album) ReplayGain tags

If that happens, go back and read about replay gain, add the tags, and try again ;).

See my replay gain post for more details on the --apply-replaygain-which-is-not-lossless option. After the decoding step, you'll have a directory full of WAVs that have been normalized to a standard track-level loudness. The bit about i and j ensures that *.wav will list the tracks in the order in which they appear in your playlist. Then burn the tracks to a CD using cdrecord:

$ cdrecord -v speed=1 dev=/dev/cdrom -eject -dao -audio -pad *.wav

If you don't care about the order, use $(ls *.wav | shuf) instead of *.wav.

That's it! Audio CDs from MPD playlists in two lines.

Posted Fri Sep 23 01:37:18 2011 Tags: linux

Available in a git repository.
Repository: cookbook
Author: W. Trevor King

I've been running a home-rolled recipe webapp for a year now, and it worked fairly well in a bare-bones sort of way. However, I recently had to make some changes to my personal website (since EveryDNS and aparently most other free DNS providers were bought by Dyn), which prompted me to translate cookbook into a Django app. Thanks to the wonders of Django, Grappelli, and django-taggit, the code is now leaner, meaner, and prettier!

See the README for details.

Posted Fri Aug 5 13:59:01 2011 Tags: linux

Available in a git repository.
Repository: h5config
Author: W. Trevor King

Since the number of packages mooching off pypiezo's configuration scheme was growing, I've split it out into it's own package. Now there's a general package for all your HDF5-based configuration needs.

The README is posted on the PyPI page.

Posted Thu Jul 28 12:25:06 2011 Tags: linux

Available in a git repository.
Repository: pypid
Author: W. Trevor King

I've just finished rewriting my PID temperature control package in pure-Python, and it's now clean enough to go up on PyPI. Features:

  • Backend-agnostic architecture. I've written a first-order process with dead time (FOPDT) test backend and a pymodbus-based backend for our Melcor MTCA controller, but it should be easy to plug in your own custom backend.
  • The general PID controller will automatically tune your backend using any of a variety of tuning rules.

The README is posted on the PyPI page.

Posted Wed Jul 27 12:06:19 2011 Tags: linux

I was suprised the the other day by a SciPy-linking issue:

$ python -c 'import scipy.linalg'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/scipy/linalg/__init__.py", line 9, in <module>
    from basic import *
  File "/usr/lib/python2.7/site-packages/scipy/linalg/basic.py", line 14, in <module>
    from lapack import get_lapack_funcs
  File "/usr/lib/python2.7/site-packages/scipy/linalg/lapack.py", line 15, in <module>
    from scipy.linalg import clapack
ImportError: /usr/lib/python2.7/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv

Searching around, it turns out there have been problems like this for a while, and it's flared up again recently. There's a new Gentoo bug tracking the most recent issues. In general, NumPy/SciPy seems to be picky about the particular BLAS implementation you're using, and you'll get problems like the above if you're using the reference implementations of BLAS and LAPACK.

$ for x in blas cblas lapack; do eselect $x list; done
Installed BLAS for library directory lib
  [1]   reference *
Installed CBLAS for library directory lib
  [1]   gsl
  [2]   reference *
Installed LAPACK for library directory lib
  [1]   reference *

You can fix the problem by installing and selecting the ATLAS libraries

$ sudo emerge -av blas-atlas lapack-atlas
$ for x in blas cblas lapack; do sudo eselect $x set atlas; done

after which you should re-install any packages that may need to be relinked.

$ sudo emerge -av numpy scipy

Now you can import the linear algebra module without crashing:

$ python -c 'import scipy.linalg'
Posted Mon Jul 25 06:55:26 2011 Tags: linux

Available in a git repository.
Repository: insider
Author: W. Trevor King

Insider is a little Django app I wrote to help my brother, Garrett, track insider trading with a simple, familiar web interface. It's a pretty simple app, partly thanks to Bradley Ayers' django-tables2, which does the table formatting. Just goes to show that a good scripting language and framework make developing simple apps a breeze!

The README is posted on the PyPI page.

Posted Thu Jul 21 18:30:33 2011 Tags: linux