Pages related to the web (HTTP, HTML, CSS, etc.).
I spent some time today configuring Postfix so I could send mail
from home via SMTPS. Verizon, our ISP, blocks port 25 to
external domains, forcing all outgoing mail through their
outgoing.verizon.net
exchange server. In order to accept mail, they
also require you authenticate with your Verizon username and password,
so I wanted to use an encrypted connection.
For the purpose of this example, our Verizon username is jdoe
, our
Verizon password is YOURPASS
, you're running a local Postfix server
on mail.example.com
for your site at example.com
, and 12345
is a
free local port.
# cat /etc/postfix/main.cf
myhostname = mail.example.com
relayhost = [127.0.0.1]:12345
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
sender_canonical_maps = hash:/etc/postfix/sender_canonical
# cat /etc/postfix/saslpass
[127.0.0.1]:12345 jdoe@verizon.net:YOURPASS
# postmap /etc/postfix/saslpass
# cat /etc/postfix/sender_canonical
root@mail.example.com jdoe@example.com
root@example.com jdoe@example.com
root@localhost jdoe@example.com
jdoe@mail.example.com jdoe@example.com
jdoe@localhost jdoe@example.com
# postmap /etc/postfix/sender_canonical
# cat /etc/stunnel/stunnel.conf
[smtp-tls-wrapper]
accept = 12345
client = yes
connect = outgoing.verizon.net:465
# /etc/init.d/stunnel restart
# postfix reload
Test with:
$ echo 'testing 1 2' | sendmail you@somewhere.com
Here's what's going on:
- You hand an outgoing message to your local Postfix, which decides to
send it via port
12345
on your localhost (127.0.0.1
) (relayhost
). - Stunnel picks up the connection from Postfix, encrypts everything,
and forwards the connection to port 465 on
outgoing.verizon.net
(stunnel.conf
). - Postfix identifies itself as
mail.example.com
(myhostname
), and authenticates using your Verizon credentials (smtp_sasl_…
). - Because Verizon is picky about the
From
addresses it will accept, we usesender_canonical
to map addresses to something simple that we've tested.
And that's it :p. If you're curious, there's more detail about all the Postfix config options in the postconf man page, and there's good SASL information in the SASL_README.
There's also a blog post by Tim White which I found useful. Because Verizon lacks STARTTLS support, his approach didn't work for me out of the box.
Verizon blocks outgoing connections on port 25 (SMTP) unless you
are connecting to their outgoing.verizon.net
message exchange
server. This server requires authentication with your Verzon
username/password before it will accept your mail. For the purpose of
this example, our Verizon username is jdoe
, our Verizon password is
YOURPASS
, and were sending email from me@example.com
to
you@target.edu
.
$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
mail from: <jdoe@example.com>
550 5.7.1 Authentication Required
quit
221 2.3.0 Bye received. Goodbye.
Because authenticating over an unencrypted connection is a Bad Idea™,
I was looking for an encrypted way to send my outgoing email.
Unfortunately, Verizon's exchange server does not support STARTTLS
for encrypting connections to outgoing.verizon.net:25
:
$ nc outgoing.verizon.net 25
220 vms173003pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173003pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP E9B7EB199A9B52CF7D936A4DD3199D6F
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
Verizon recommends pre-STARTTLS approach of wrapping the
whole SMTP connection in TLS (SMTPS), which it provides via
outgoing.verizon.net:465
:
$ python -c 'from base64 import *; print b64encode("\0jdoe@verizon.net\0YOURPASS")'
AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M=
$ openssl s_client -connect outgoing.verizon.net:465
...
220 vms173013pub.verizon.net -- Server ESMTP (...)
ehlo example.com
250-vms173013pub.verizon.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-HELP
250-XLOOP 9380A5843FE933CF9BD037667F4C950D
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 20971520
auth plain AGpkb2VAdmVyaXpvbi5uZXQAWU9VUlBBU1M
235 2.7.0 plain authentication successful.
mail from: <me@example.com>
250 2.5.0 Address Ok.
rcpt to: <you@target.edu>
250 2.1.5 you@target.edu OK.
data
354 Enter mail, end with a single ".".
From: Me <me@example.com>
To: You <you@target.edu>
Subject: testing
hello world
.
250 2.5.0 Ok, envelope id 4BHMFEZ7PHSETMT6@vms173013.mailsrvcs.net
quit
221 2.3.0 Bye received. Goodbye.
closed
This works, but with the rise of STARTTLS, getting your local Postfix mail server to support SMTPS requires a bit of fancyness with stunnel. The stunnel workaround is not too complicated, but I also wanted to look into the submission protocol (port 587), which adapts SMTP (designed for message transfer) into a similar protocol for message submission. Unfortunately, Verizon does not support STARTTLS here either.
$ nc outgoing.verizon.net 587
220 vms173005.mailsrvcs.net -- Server ESMTP (...)
ehlo example.com
250-vms173005.mailsrvcs.net
250-8BITMIME
250-PIPELINING
250-CHUNKING
250-DSN
250-ENHANCEDSTATUSCODES
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP DA941C5B31BE4B102BB69B809BC66C4A
250-AUTH DIGEST-MD5 PLAIN LOGIN CRAM-MD5
250-AUTH=LOGIN PLAIN
250-NO-SOLICITING
250 SIZE 20971520
starttls
533 5.7.1 STARTTLS command is not enabled.
quit
221 2.3.0 Bye received. Goodbye.
In conclusion, Verizon supports a number of email submission standards, but the only secure approach is to use the outdated SMTPS. See my Postfix post for details on configuring Postfix to use Verizon's server for outgoing mail.
There are a number of good SMTP authentication tutorials out there. I
used John Simpson and Erwin Hoffmann's tutorials. For
cleaner examples of my testing tools (nc
and openssl s_client
),
see my simple servers post.
I generally write my presentations up in LaTeX and use beamer to convert them to PDFs for display. I don't really like PDFs though, they're big and unwieldy. I was just looking around at alternative methods and ran across the older Linux for Presentations Mini-HOWTO which lists a number of HTML-based presentation formats. This is definately the way to go, and with MathML (via itex2MML, see my mdwn itex post for details) I can generate most of the equations that I need to fit into a standard presentation.
The two grandaddy frameworks are S5 and Slidy. S5 goes back to 2004, and Slidy has been around since 2005, so they both go back a ways. The code you'll be writing is fairly similar in each of them, so just pick whichever you like best. If it matters to you, Slidy is a W3C recommendation
However, neither of these early projects seem to be actively developed. For example, this thread on the S5 Project Google group discusses fracturing in the S5 community. As people try to work around issues with the older frameworks, they've written up new frameworks. S6 wiki has some pointers to other options.
Personally, I'm going to go with S6, since the project uses Git for version control which is a Good Thing. There's also S9, which lets you build S6 presentations with some sort of wiki-syntax instead of HTML, if that's appealing to you. If you need MathML on Firefoxes before version 4 (which includes HTML 5 support), you'll need a framework like Slidy that supports XHTML.
Finally, there's a webslideshow Google group that tracks developments in this area.
Good luck!
The Drexel physics department moved most of its content off of the department servers and onto college servers this quarter. The college servers manage their content with SiteCore, so there was a reasonable amount of trouble getting everything over (see SiteCorePy). Luckily, I got lots of help, and now I don't have to worry about the content that has migrated :). However, not all of the content made the switch.
We have a number of forms and databases that stayed on our department servers, and it's my job to make sure those pages look similar to the SiteCore pages that link to them. No problem, you say, just clone the SiteCore page's CSS, and apply it to the local pages. That's exactly what I want to do, but the jittery folks upstream keep changing the CSS, so my cloned CSS gets out of sync fairly quickly. To minimize my suffering, I've written a little script to automate the task of cloning another page's CSS.
get css.py scrapes an (X)HTML page for stylesheets (assuming there
is no embedded styling in the HTML itself). It then downloads all
those CSS files, cleans them up with cssutils, and saves a single
clone stylesheet mimicking their behaviour. It also downloads all
media referenced via url(...)
entries in the CSS (e.g. background
images), and adjusts the CSS to point to the local copies.
FOAF is an RDF language describing social networks and related information. FOAF tries to social networking available to the Semantic Web, making your identity and friendships machine readable. An interesting FOAF application, if you have enough tech-savy friends is Foaf+SSL. For an intuitive explanation of how FOAF+SSL works, take a look at Henry Story's paradigm shift post.
For the curious, I've created my own FOAF file. You should also check out the SIOC Project's Semantic Radar plugin, which notifies you of the existence of interesting RDF data as you browse the web (and inspired my rel-vcs plugin).
Available in a git repository.
Repository: parallel_computing
Author: W. Trevor King
In contrast to my course website project, which is mostly about constructing a framework for automatically compiling and installing LaTeX problem sets, Prof. Vallières' Parallel Computing course is basically an online textbook with a large amount of example software. In order to balance between to Prof. Vallières' original and my own aesthetic, I rolled a new solution from scratch. See my version of his Fall 2010 page for a live example.
Differences from my course website project:
- No PHP, since there is no dynamic content that cannot be handled with SSI.
- Less installation machinery. Only a few build/cleanup scripts to
avoid versioning really tedious bits. The repository is designed to
be dropped into your
~/public_html/
whole, while the course website project is designed torsync
the built components up as they go live. - Less LaTeX, more XHTML. It's easier to edit XHTML than it is to exit and compile LaTeX, and PDFs are large and annoying. As a computing class, there are fewer graphics than there are in an intro-physics class, so the extra power of LaTeX is not as useful.
Available in a git repository.
Repository: intro-physics
Author: W. Trevor King
Over a few years as a TA for assorted introductory physics classes,
I've assembled a nice website framework with lots of problems using my
LaTeX problempack package, along with some handy Makefiles
,
a bit of php, and SSI.
The result is the intro-physics
package, which should make it very
easy to whip up a course website, homeworks, etc. for an introductory
mechanics or E&M class (321 problems implemented as of October 2010).
With a bit of work to write up problems, the framework could easily be
extended to other subjects.
The idea is that a course website consists of a small, static HTML
framework, and a bunch of content that is gradually filled in as the
semester/quarter progresses. I've put the HTML framework in the
html/
directory, along with some of the write-once-per-course
content (e.g. Prof & TA info). See html/README
for more information
on the layout of the HTML.
The rest of the directories contain the code for compiling material
that is deployed as the course progresses. The announcements/
directory contains the atom feed for the course, and possibly a list
of email addresses of people who would like to (or should) be notified
when new announcements are posted. The latex/
directory contains
LaTeX source for the course documents for which it is available, and
the pdf/
directory contains PDFs for which no other source is
available (e.g. scans, or PDFs sent in by Profs or TAs who neglected
to include their source code).
Note that because this framework assumes the HTML content will be relatively static, it may not be appropriate for courses with large amounts of textbook-style content, which will undergo more frequent revision. It mayq also be excessive for courses that need less compiled content. For an example of another framework, see my branch of Prof. Vallières' Parallel Computing website.
Since I publish a lot of Git packages, I was interested to read about Joey Hess' rel=vcs-* microformat. I think recording the location of the repo sorcing a page is a great idea, but with the link stashed in the page header, I could easily browse on by without ever noticing that the link existed.
This looks like the same sort of problem that the Semantic Radar
extension was designed to solve, except the SR extension notifies you
about RDF files (SIOC, FOAF, DOAP, etc.). I've altered the SR
extension to identify the rel=vcs-*
tags.
The rel-vcs extension places an icon in your Firefox
statusbar which goes "hot" when a page has rel=vcs-*
tags and "cold"
otherwise. When the icon is "hot", you can click on it to pop up a
list of rel-vcs links. Clicking on an item in the list will open that
URI in a new tab. Since Firefox can't speak git://
etc., the new
tab will mostly be useful as a source of the URI for copy/pasting into
a git clone ...
call or similar. Alternatively, you can consider
the "hot" icon as a suggestion to use webcheckout
or other
rel=vcs-*
consumer on the source page.
Literate programming is a philosophy of computer programming based on the premise that a computer program should be written similar to literature, with human readability as a primary goal.
Traditional programs have human language comments interspersed in computer language code. Literate programs reverse this style, with computer language "comments" interspersed in a human language essay.
An excellent (and simple) literate programming tool is noweb. There is a hello-world example and some intro information on Wikipedia. There are also official quick and 6 page introductions.
Often data is stored in XML files must be massaged into other formats (e.g. DocBook to roff). There are well developed procedures for defining such transformations (XSLT) and a number of tools to apply them (e.g. xsltproc).
Besides the W3 tutorial, there is also a nice introduction by Paul Grosso and Norman Walsh. I've copied a simple example from this intro and also included a slightly more complicated setup for generating online help for a list of macros.
XSLT is also useful for standardizing XML content. For example, I was
recently trying to compare to Gramps XML files, to see what had
changed between two database backups. Unfortunately, the backup XML
was not sorted by id
, so there were many diff chunks due to node
shuffling that didn't represent any useful information. With the
following XSLT:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*"/>
<!-- identity transform -->
<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<!-- sort node children by their `id` attributes -->
<xsl:template match="node()">
<xsl:copy>
<xsl:apply-templates select="@*"/>
<xsl:for-each select="node()">
<xsl:sort select="@id" order="ascending"/>
<xsl:apply-templates select="."/>
</xsl:for-each>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
With the above saved as sort-by-id.xsl
, you can sort some.xml
using
$ xsltproc --nonet --novalid sort-by-id.xsl some.xml
You can compare two Gramps XML files with
$ diff -u <(zcat a.gramps | xsltproc --nonet --novalid sort-by-id.xsl -)
<(zcat b.gramps | xsltproc --nonet --novalid sort-by-id.xsl -) | less
Jesper Tverskov has a nice page about the identity template and related tricks if you want more examples of quasi-copy transforms.