Linux-related pages, mostly about system administration.
Mailcap files, defined in RFC 1524, allow you to tell you mail clients, web browsers, and other programs how you want to view and edit various MIME types. Since interaction with your mailcap files often occurs deep within the bowels of your program, I've written up a very simple Python script to test your mailcap entries: mailcap-test.py. Enjoy!
I spent some time today configuring Postfix so I could send mail
from home via SMTPS. Verizon, our ISP, blocks port 25 to
external domains, forcing all outgoing mail through their
outgoing.verizon.net
exchange server. In order to accept mail, they
also require you authenticate with your Verizon username and password,
so I wanted to use an encrypted connection.
For the purpose of this example, our Verizon username is jdoe
, our
Verizon password is YOURPASS
, you're running a local Postfix server
on mail.example.com
for your site at example.com
, and 12345
is a
free local port.
# cat /etc/postfix/main.cf
myhostname = mail.example.com
relayhost = [127.0.0.1]:12345
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
sender_canonical_maps = hash:/etc/postfix/sender_canonical
# cat /etc/postfix/saslpass
[127.0.0.1]:12345 jdoe@verizon.net:YOURPASS
# postmap /etc/postfix/saslpass
# cat /etc/postfix/sender_canonical
root@mail.example.com jdoe@example.com
root@example.com jdoe@example.com
root@localhost jdoe@example.com
jdoe@mail.example.com jdoe@example.com
jdoe@localhost jdoe@example.com
# postmap /etc/postfix/sender_canonical
# cat /etc/stunnel/stunnel.conf
[smtp-tls-wrapper]
accept = 12345
client = yes
connect = outgoing.verizon.net:465
# /etc/init.d/stunnel restart
# postfix reload
Test with:
$ echo 'testing 1 2' | sendmail you@somewhere.com
Here's what's going on:
- You hand an outgoing message to your local Postfix, which decides to
send it via port
12345
on your localhost (127.0.0.1
) (relayhost
). - Stunnel picks up the connection from Postfix, encrypts everything,
and forwards the connection to port 465 on
outgoing.verizon.net
(stunnel.conf
). - Postfix identifies itself as
mail.example.com
(myhostname
), and authenticates using your Verizon credentials (smtp_sasl_…
). - Because Verizon is picky about the
From
addresses it will accept, we usesender_canonical
to map addresses to something simple that we've tested.
And that's it :p. If you're curious, there's more detail about all the Postfix config options in the postconf man page, and there's good SASL information in the SASL_README.
There's also a blog post by Tim White which I found useful. Because Verizon lacks STARTTLS support, his approach didn't work for me out of the box.
Verizon blocks outgoing connections on port 25 (SMTP) unless you
are connecting to their outgoing.verizon.net
message exchange
server. This server requires authentication with your Verzon
username/password before it will accept your mail. For the purpose of
this example, our Verizon username is jdoe
, our Verizon password is
YOURPASS
, and were sending email from me@example.com
to
you@target.edu
.
$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
mail from: <jdoe@example.com>
550 5.7.1 Authentication Required
quit
221 2.3.0 Bye received. Goodbye.
Because authenticating over an unencrypted connection is a Bad Idea™,
I was looking for an encrypted way to send my outgoing email.
Unfortunately, Verizon's exchange server does not support STARTTLS
for encrypting connections to outgoing.verizon.net:25
:
$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173003pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP E9B7EB199A9B52CF7D936A4DD3199D6F
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
Verizon recommends pre-STARTTLS approach of wrapping the
whole SMTP connection in TLS (SMTPS), which it provides via
outgoing.verizon.net:465
:
$ python -c 'from base64 import *; print b64encode("\0jdoe@verizon.net\0YOURPASS")'
AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M=
$ openssl s_client -connect outgoing.verizon.net:465
...
220 vms173013pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173013pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP 9380A5843FE933CF9BD037667F4C950D
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
auth plain AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M
235 2.7.0 plain authentication successful.
mail from: <me@example.com>
250 2.5.0 Address Ok.
rcpt to: <you@target.edu>
250 2.1.5 you@target.edu OK.
data
354 Enter mail, end with a single ".".
From: Me <me@example.com>
To: You <you@target.edu>
Subject: testing
hello world
.
250 2.5.0 Ok, envelope id 4BHMFEZ7PHSETMT6@vms173013.mailsrvcs.net
quit
221 2.3.0 Bye received. Goodbye.
closed
This works, but with the rise of STARTTLS, getting your local Postfix mail server to support SMTPS requires a bit of fancyness with stunnel. The stunnel workaround is not too complicated, but I also wanted to look into the submission protocol (port 587), which adapts SMTP (designed for message transfer) into a similar protocol for message submission. Unfortunately, Verizon does not support STARTTLS here either.
$ nc outgoing.verizon.net 587
220 vms173005.mailsrvcs.net -- Server ESMTP (...)
ehlo example.com
250-vms173005.mailsrvcs.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP DA941C5B31BE4B102BB69B809BC66C4A
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
In conclusion, Verizon supports a number of email submission standards, but the only secure approach is to use the outdated SMTPS. See my Postfix post for details on configuring Postfix to use Verizon's server for outgoing mail.
There are a number of good SMTP authentication tutorials out there. I
used John Simpson and Erwin Hoffmann's tutorials. For
cleaner examples of my testing tools (nc
and openssl s_client
),
see my simple servers post.
I was recently trying to add bookmarks to a PDF I'd generated with pdftk. It turns out to be fairly simple to add bookmarks to a PDF using Ghostscript, following maggoteer's post to the Ubunto forums. The syntax is:
$ gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=out.pdf in-*.pdf pdfmarks
Where out.pdf
is the generated PDF, in-*.pdf
are the input PDFs,
and pdfmarks
is a text file with contents like:
[/Title (Title Page) /Page 1 /OUT pdfmark
[/Title (Table of Contents) /Page 3 /OUT pdfmark
...
Nice and easy.
For nested levels, use the /Count
attribute. For example:
[/Count 3 /Title (Chapter 1) /Page 1 /OUT pdfmark
[/Count -2 /Title (Section 1.1) /Page 2 /OUT pdfmark
[/Title (Section 1.1.1) /Page 3 /OUT pdfmark
[/Title (Section 1.1.2) /Page 4 /OUT pdfmark
[/Count -1 /Title (Section 1.2) /Page 5 /OUT pdfmark
[/Title (Section 1.2.1) /Page 6 /OUT pdfmark
[/Title (Section 1.3) /Page 7 /OUT pdfmark
The argument to /Count
gives the number of immediately subordinate
bookmarks. The sign of the argument sets the default display
(negative for closed, positive for open).
You can also setup the document info dictionary with something like:
[ /Title (My Test Document)
/Author (John Doe)
/Subject (pdfmark 3.0)
/Keywords (pdfmark, example, test)
/DOCINFO pdfmark
If you want more detail, take a look at Adobe's pdfmark reference.
I've bundled the whole pdfmarks-generation bit into a script,
pdf-merge.py, which generates the pdfmark file and runs
Ghostscript automatically. Think of it as a bookmark-preserving
version of pdftk's cat
. The script uses pdftk internally to extract
bookmark information from the source PDFs.
The script also adds a bit of PostScript to ignore any bookmarks in
the source PDFs during the Ghostscript run. The only bookmarks in the
output will be the ones you specify explicitly in the pdfmarks file.
If for some reason the automatically generated pdfmarks are not quite
what you want, the script can pause (via --ask
) to allow you to
tweak the pdfmarks manually before running Ghostscript.
I've been streamlining my procedure for burning audio CDs, and I like what I've come up with. Unfortunately, I'll never remember it on my own, so here's a note to myself (any whoever else cares) on what to do.
First, build your playlist in MPD, using my one-liner to calculate the playlist duration (my CD-Rs hold 80 minutes). When you've got your playlist arranged to your satisfaction, convert the files to WAVs using:
$ i=1; for x in $(mpc -f '%file%' playlist); do j=$(seq -f '%02g' $i $i); let "i += 1"; flac -d --apply-replaygain-which-is-not-lossless=t -o $j.$(basename ${x/.flac/.wav}) /var/lib/mpd/music/$x; done
This assumes that all the files in your playlist are FLAC files, which is a good idea (disk space is cheap, FLAC is lossless with good open source support). It also assumes you've already stored ReplayGain settings in your FLAC files. If you haven't, you'll get warnings like:
WARNING: can't get track (or even album) ReplayGain tags
If that happens, go back and read about replay gain, add the tags, and try again ;).
See my replay gain post for more details on the
--apply-replaygain-which-is-not-lossless
option. After the decoding
step, you'll have a directory full of WAVs that have been normalized
to a standard track-level loudness. The bit about i
and j
ensures
that *.wav
will list the tracks in the order in which they appear in
your playlist. Then burn the tracks to a CD using
cdrecord:
$ cdrecord -v speed=1 dev=/dev/cdrom -eject -dao -audio -pad *.wav
If you don't care about the order, use $(ls *.wav | shuf)
instead of
*.wav
.
That's it! Audio CDs from MPD playlists in two lines.
Available in a git repository.
Repository: cookbook
Author: W. Trevor King
I've been running a home-rolled recipe webapp for a year now, and it
worked fairly well in a bare-bones sort of way. However, I recently
had to make some changes to my personal website (since EveryDNS and
aparently most other free DNS providers were bought by Dyn), which
prompted me to translate cookbook
into a Django app. Thanks to
the wonders of Django, Grappelli, and django-taggit, the code
is now leaner, meaner, and prettier!
See the README for details.
Available in a git repository.
Repository: h5config
Author: W. Trevor King
Since the number of packages mooching off pypiezo's configuration scheme was growing, I've split it out into it's own package. Now there's a general package for all your HDF5-based configuration needs.
The README
is posted on the PyPI page.
Available in a git repository.
Repository: pypid
Author: W. Trevor King
I've just finished rewriting my PID temperature control package in pure-Python, and it's now clean enough to go up on PyPI. Features:
- Backend-agnostic architecture. I've written a first-order process with dead time (FOPDT) test backend and a pymodbus-based backend for our Melcor MTCA controller, but it should be easy to plug in your own custom backend.
- The general PID controller will automatically tune your backend using any of a variety of tuning rules.
The README
is posted on the PyPI page.
I was suprised the the other day by a SciPy-linking issue:
$ python -c 'import scipy.linalg'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/scipy/linalg/__init__.py", line 9, in <module>
from basic import *
File "/usr/lib/python2.7/site-packages/scipy/linalg/basic.py", line 14, in <module>
from lapack import get_lapack_funcs
File "/usr/lib/python2.7/site-packages/scipy/linalg/lapack.py", line 15, in <module>
from scipy.linalg import clapack
ImportError: /usr/lib/python2.7/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv
Searching around, it turns out there have been problems like this for a while, and it's flared up again recently. There's a new Gentoo bug tracking the most recent issues. In general, NumPy/SciPy seems to be picky about the particular BLAS implementation you're using, and you'll get problems like the above if you're using the reference implementations of BLAS and LAPACK.
$ for x in blas cblas lapack; do eselect $x list; done
Installed BLAS for library directory lib
[1] reference *
Installed CBLAS for library directory lib
[1] gsl
[2] reference *
Installed LAPACK for library directory lib
[1] reference *
You can fix the problem by installing and selecting the ATLAS libraries
$ sudo emerge -av blas-atlas lapack-atlas
$ for x in blas cblas lapack; do sudo eselect $x set atlas; done
after which you should re-install any packages that may need to be relinked.
$ sudo emerge -av numpy scipy
Now you can import the linear algebra module without crashing:
$ python -c 'import scipy.linalg'
Available in a git repository.
Repository: insider
Author: W. Trevor King
Insider is a little Django app I wrote to help my brother, Garrett, track insider trading with a simple, familiar web interface. It's a pretty simple app, partly thanks to Bradley Ayers' django-tables2, which does the table formatting. Just goes to show that a good scripting language and framework make developing simple apps a breeze!
The README
is posted on the PyPI page.