Pages related to the web (HTTP, HTML, CSS, etc.).

I spent some time today configuring Postfix so I could send mail from home via SMTPS. Verizon, our ISP, blocks port 25 to external domains, forcing all outgoing mail through their outgoing.verizon.net exchange server. In order to accept mail, they also require you authenticate with your Verizon username and password, so I wanted to use an encrypted connection.

For the purpose of this example, our Verizon username is jdoe, our Verizon password is YOURPASS, you're running a local Postfix server on mail.example.com for your site at example.com, and 12345 is a free local port.

# cat /etc/postfix/main.cf
myhostname = mail.example.com
relayhost = [127.0.0.1]:12345
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
sender_canonical_maps = hash:/etc/postfix/sender_canonical
# cat /etc/postfix/saslpass
[127.0.0.1]:12345 jdoe@verizon.net:YOURPASS
    # postmap /etc/postfix/saslpass
    # cat /etc/postfix/sender_canonical
    root@mail.example.com jdoe@example.com
root@example.com jdoe@example.com
root@localhost jdoe@example.com
jdoe@mail.example.com jdoe@example.com
jdoe@localhost jdoe@example.com
    # postmap /etc/postfix/sender_canonical
# cat /etc/stunnel/stunnel.conf
[smtp-tls-wrapper]
accept = 12345
client = yes
connect = outgoing.verizon.net:465
# /etc/init.d/stunnel restart
# postfix reload

Test with:

$ echo 'testing 1 2' | sendmail you@somewhere.com

Here's what's going on:

  • You hand an outgoing message to your local Postfix, which decides to send it via port 12345 on your localhost (127.0.0.1) (relayhost).
  • Stunnel picks up the connection from Postfix, encrypts everything, and forwards the connection to port 465 on outgoing.verizon.net (stunnel.conf).
  • Postfix identifies itself as mail.example.com (myhostname), and authenticates using your Verizon credentials (smtp_sasl_…).
  • Because Verizon is picky about the From addresses it will accept, we use sender_canonical to map addresses to something simple that we've tested.

And that's it :p. If you're curious, there's more detail about all the Postfix config options in the postconf man page, and there's good SASL information in the SASL_README.

There's also a blog post by Tim White which I found useful. Because Verizon lacks STARTTLS support, his approach didn't work for me out of the box.

Posted Tue Nov 15 19:48:33 2011 Tags: web

Verizon blocks outgoing connections on port 25 (SMTP) unless you are connecting to their outgoing.verizon.net message exchange server. This server requires authentication with your Verzon username/password before it will accept your mail. For the purpose of this example, our Verizon username is jdoe, our Verizon password is YOURPASS, and were sending email from me@example.com to you@target.edu.

$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
mail from: <jdoe@example.com>  
550 5.7.1 Authentication Required
quit
221 2.3.0 Bye received. Goodbye.

Because authenticating over an unencrypted connection is a Bad Idea™, I was looking for an encrypted way to send my outgoing email. Unfortunately, Verizon's exchange server does not support STARTTLS for encrypting connections to outgoing.verizon.net:25:

$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173003pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP E9B7EB199A9B52CF7D936A4DD3199D6F
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.

Verizon recommends pre-STARTTLS approach of wrapping the whole SMTP connection in TLS (SMTPS), which it provides via outgoing.verizon.net:465:

$ python -c 'from base64 import *; print b64encode("\0jdoe@verizon.net\0YOURPASS")'
AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M=
$ openssl s_client -connect outgoing.verizon.net:465
...
220 vms173013pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173013pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP 9380A5843FE933CF9BD037667F4C950D
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
auth plain AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M
235 2.7.0 plain authentication successful.
mail from: <me@example.com>
250 2.5.0 Address Ok.
rcpt to: <you@target.edu>
250 2.1.5 you@target.edu OK.
data
354 Enter mail, end with a single ".".
From: Me <me@example.com>
To: You <you@target.edu>
Subject: testing

hello world 
.
250 2.5.0 Ok, envelope id 4BHMFEZ7PHSETMT6@vms173013.mailsrvcs.net
quit
221 2.3.0 Bye received. Goodbye.
closed

This works, but with the rise of STARTTLS, getting your local Postfix mail server to support SMTPS requires a bit of fancyness with stunnel. The stunnel workaround is not too complicated, but I also wanted to look into the submission protocol (port 587), which adapts SMTP (designed for message transfer) into a similar protocol for message submission. Unfortunately, Verizon does not support STARTTLS here either.

$ nc outgoing.verizon.net 587
220 vms173005.mailsrvcs.net -- Server ESMTP (...)
ehlo example.com
250-vms173005.mailsrvcs.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP DA941C5B31BE4B102BB69B809BC66C4A
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.

In conclusion, Verizon supports a number of email submission standards, but the only secure approach is to use the outdated SMTPS. See my Postfix post for details on configuring Postfix to use Verizon's server for outgoing mail.

There are a number of good SMTP authentication tutorials out there. I used John Simpson and Erwin Hoffmann's tutorials. For cleaner examples of my testing tools (nc and openssl s_client), see my simple servers post.

Posted Tue Nov 15 19:45:35 2011 Tags: web

I generally write my presentations up in LaTeX and use beamer to convert them to PDFs for display. I don't really like PDFs though, they're big and unwieldy. I was just looking around at alternative methods and ran across the older Linux for Presentations Mini-HOWTO which lists a number of HTML-based presentation formats. This is definately the way to go, and with MathML (via itex2MML, see my mdwn itex post for details) I can generate most of the equations that I need to fit into a standard presentation.

The two grandaddy frameworks are S5 and Slidy. S5 goes back to 2004, and Slidy has been around since 2005, so they both go back a ways. The code you'll be writing is fairly similar in each of them, so just pick whichever you like best. If it matters to you, Slidy is a W3C recommendation

However, neither of these early projects seem to be actively developed. For example, this thread on the S5 Project Google group discusses fracturing in the S5 community. As people try to work around issues with the older frameworks, they've written up new frameworks. S6 wiki has some pointers to other options.

Personally, I'm going to go with S6, since the project uses Git for version control which is a Good Thing. There's also S9, which lets you build S6 presentations with some sort of wiki-syntax instead of HTML, if that's appealing to you. If you need MathML on Firefoxes before version 4 (which includes HTML 5 support), you'll need a framework like Slidy that supports XHTML.

Finally, there's a webslideshow Google group that tracks developments in this area.

Good luck!

Posted Tue May 3 07:02:25 2011 Tags: web

The Drexel physics department moved most of its content off of the department servers and onto college servers this quarter. The college servers manage their content with SiteCore, so there was a reasonable amount of trouble getting everything over (see SiteCorePy). Luckily, I got lots of help, and now I don't have to worry about the content that has migrated :). However, not all of the content made the switch.

We have a number of forms and databases that stayed on our department servers, and it's my job to make sure those pages look similar to the SiteCore pages that link to them. No problem, you say, just clone the SiteCore page's CSS, and apply it to the local pages. That's exactly what I want to do, but the jittery folks upstream keep changing the CSS, so my cloned CSS gets out of sync fairly quickly. To minimize my suffering, I've written a little script to automate the task of cloning another page's CSS.

get css.py scrapes an (X)HTML page for stylesheets (assuming there is no embedded styling in the HTML itself). It then downloads all those CSS files, cleans them up with cssutils, and saves a single clone stylesheet mimicking their behaviour. It also downloads all media referenced via url(...) entries in the CSS (e.g. background images), and adjusts the CSS to point to the local copies.

Posted Sun Jan 9 07:24:31 2011 Tags: web

FOAF is an RDF language describing social networks and related information. FOAF tries to social networking available to the Semantic Web, making your identity and friendships machine readable. An interesting FOAF application, if you have enough tech-savy friends is Foaf+SSL. For an intuitive explanation of how FOAF+SSL works, take a look at Henry Story's paradigm shift post.

For the curious, I've created my own FOAF file. You should also check out the SIOC Project's Semantic Radar plugin, which notifies you of the existence of interesting RDF data as you browse the web (and inspired my rel-vcs plugin).

Posted Tue Nov 9 12:39:59 2010 Tags: web

Available in a git repository.
Repository: parallel_computing
Author: W. Trevor King

In contrast to my course website project, which is mostly about constructing a framework for automatically compiling and installing LaTeX problem sets, Prof. Vallières' Parallel Computing course is basically an online textbook with a large amount of example software. In order to balance between to Prof. Vallières' original and my own aesthetic, I rolled a new solution from scratch. See my version of his Fall 2010 page for a live example.

Differences from my course website project:

  • No PHP, since there is no dynamic content that cannot be handled with SSI.
  • Less installation machinery. Only a few build/cleanup scripts to avoid versioning really tedious bits. The repository is designed to be dropped into your ~/public_html/ whole, while the course website project is designed to rsync the built components up as they go live.
  • Less LaTeX, more XHTML. It's easier to edit XHTML than it is to exit and compile LaTeX, and PDFs are large and annoying. As a computing class, there are fewer graphics than there are in an intro-physics class, so the extra power of LaTeX is not as useful.
Posted Thu Oct 14 08:59:13 2010 Tags: web

Available in a git repository.
Repository: intro-physics
Author: W. Trevor King

Over a few years as a TA for assorted introductory physics classes, I've assembled a nice website framework with lots of problems using my LaTeX problempack package, along with some handy Makefiles, a bit of php, and SSI.

The result is the intro-physics package, which should make it very easy to whip up a course website, homeworks, etc. for an introductory mechanics or E&M class (321 problems implemented as of October 2010). With a bit of work to write up problems, the framework could easily be extended to other subjects.

The idea is that a course website consists of a small, static HTML framework, and a bunch of content that is gradually filled in as the semester/quarter progresses. I've put the HTML framework in the html/ directory, along with some of the write-once-per-course content (e.g. Prof & TA info). See html/README for more information on the layout of the HTML.

The rest of the directories contain the code for compiling material that is deployed as the course progresses. The announcements/ directory contains the atom feed for the course, and possibly a list of email addresses of people who would like to (or should) be notified when new announcements are posted. The latex/ directory contains LaTeX source for the course documents for which it is available, and the pdf/ directory contains PDFs for which no other source is available (e.g. scans, or PDFs sent in by Profs or TAs who neglected to include their source code).

Note that because this framework assumes the HTML content will be relatively static, it may not be appropriate for courses with large amounts of textbook-style content, which will undergo more frequent revision. It mayq also be excessive for courses that need less compiled content. For an example of another framework, see my branch of Prof. Vallières' Parallel Computing website.

Posted Thu Oct 14 08:55:17 2010 Tags: web

Since I publish a lot of Git packages, I was interested to read about Joey Hess' rel=vcs-* microformat. I think recording the location of the repo sorcing a page is a great idea, but with the link stashed in the page header, I could easily browse on by without ever noticing that the link existed.

This looks like the same sort of problem that the Semantic Radar extension was designed to solve, except the SR extension notifies you about RDF files (SIOC, FOAF, DOAP, etc.). I've altered the SR extension to identify the rel=vcs-* tags.

The rel-vcs extension places an icon in your Firefox statusbar which goes "hot" when a page has rel=vcs-* tags and "cold" otherwise. When the icon is "hot", you can click on it to pop up a list of rel-vcs links. Clicking on an item in the list will open that URI in a new tab. Since Firefox can't speak git:// etc., the new tab will mostly be useful as a source of the URI for copy/pasting into a git clone ... call or similar. Alternatively, you can consider the "hot" icon as a suggestion to use webcheckout or other rel=vcs-* consumer on the source page.

Posted Thu Oct 7 23:20:04 2010 Tags: web

Literate programming is a philosophy of computer programming based on the premise that a computer program should be written similar to literature, with human readability as a primary goal.

Traditional programs have human language comments interspersed in computer language code. Literate programs reverse this style, with computer language "comments" interspersed in a human language essay.

An excellent (and simple) literate programming tool is noweb. There is a hello-world example and some intro information on Wikipedia. There are also official quick and 6 page introductions.

Posted Tue Oct 5 15:24:12 2010 Tags: web

Often data is stored in XML files must be massaged into other formats (e.g. DocBook to roff). There are well developed procedures for defining such transformations (XSLT) and a number of tools to apply them (e.g. xsltproc).

Besides the W3 tutorial, there is also a nice introduction by Paul Grosso and Norman Walsh. I've copied a simple example from this intro and also included a slightly more complicated setup for generating online help for a list of macros.

XSLT is also useful for standardizing XML content. For example, I was recently trying to compare to Gramps XML files, to see what had changed between two database backups. Unfortunately, the backup XML was not sorted by id, so there were many diff chunks due to node shuffling that didn't represent any useful information. With the following XSLT:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" indent="yes"/>
    <xsl:strip-space elements="*"/>
    <!-- identity transform -->
    <xsl:template match="@*|node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>
    <!-- sort node children by their `id` attributes -->
    <xsl:template match="node()">
        <xsl:copy>
            <xsl:apply-templates select="@*"/>
            <xsl:for-each select="node()">
                <xsl:sort select="@id" order="ascending"/>
                <xsl:apply-templates select="."/>
            </xsl:for-each>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

With the above saved as sort-by-id.xsl, you can sort some.xml using

$ xsltproc --nonet --novalid sort-by-id.xsl some.xml

You can compare two Gramps XML files with

$ diff -u <(zcat a.gramps | xsltproc --nonet --novalid sort-by-id.xsl -)
          <(zcat b.gramps | xsltproc --nonet --novalid sort-by-id.xsl -) | less

Jesper Tverskov has a nice page about the identity template and related tricks if you want more examples of quasi-copy transforms.

Posted Tue Oct 5 15:18:38 2010 Tags: web