Brand Name Necrophilia

This week, I resurrected a project that I first started in 1995 when I realized "this Internet thing is going to be HUGE". At the time, I figured one day every advertisement would feature a URL. I never realised the Internet would elect its own world leader only half a generation later.

The name for what Wikipedia describes as a "multi-year coding frenzy" was taken at the last minute in 1999 from an "iMatix" t-shirt viewed in the mirror. Wikipedia says that "the last Xitami release, 2.5 has been in beta since 1999". We abandoned Xitami/2 because it was too much work and did not generate any income. Peoples has to eat.

But times change, and the meta-programming tools (iCL, Base2, XNF) we use to make OpenAMQ and Zyre are really much easier to use than what we had in 1999.

Since Zyre needs a reliable embedded web server, and it's easier to write one from scratch than integrate an existing one, Xitami is alive again. This is what my friend Mato calls "brand name necrophilia" but I see it more like re-enacting a historical drama, or recreating a classic movie, but with shorter skirts and more explosions.

Xitami/3 did exist briefly, but the main reason for calling this new, written from scratch beast "Xitami/5" is the same reason one of my SoftToys video game titles from 1986 was called "Star Warp II". The higher version number makes people think it's a hit series. Clever marketing, see?

As I write this, Xitami's spanking new Digest Authentication module is telling me it still can't correctly calculate the MD5 hash for "MD5 (HA1 : nonce : nonceCount : clientNonce : qop : HA2)". I've been working since midday on the Digest authentication module (the Basic authentication was relatively easy). And it complains:

have:b646519251c16f971246593efca064c2 need:60fecb06726d88ea5f015f6765c34c50

Which is why some people classify security programming as "mindless masochism". It is definitely hard. Making Xitami work with Apache-format passwd and digest files is more like digging a ditch to precise specifications, using a spoon, than leaping from an exploding train wreck.

Still, there are pleasures in writing new code. Xitami/5 takes a cynical and distrustful view of the Internet. If a browser is not absolutely well-behaved, it believes, there is a crook, hacker, spammer, or idiot behind that keyboard. So, Xitami/5 has a policy language that lets me write gems like this (and XML shines for this kind of instant language):

<!-- Detect hostile requests, auto-ban offending IP addresses -->
<policy name = "auto-ban">
    <!-- Attempt to smash the server with long requests -->
    <detect limit = "255"       comment = "long request line" />
    <!-- Attempts at injections via the URI -->
    <detect value = "%3Cscript" comment = "script injection" />
    <detect value = "%3Cform"   comment = "form injection" />
    <detect value = "%20or"     comment = "SQL injection" />
    <detect value = "%20and"    comment = "SQL injection" />
    <detect value = "%20select" comment = "SQL injection" />
    <detect value = "%20drop"   comment = "SQL injection" />
    <!-- Attempts to navigate outside the web root -->
    <detect value = ".."        comment = "path climbing" />
    <detect value = "%5c"       comment = "Win32 paths" />
    <detect value = "~"         comment = "Unix paths" />
    <!-- Probe to see if we're a proxy server -->
    <detect value = "http://"   comment = "proxy probe" />
    <default>
        <echo>W: hostile request from $from ($comment), blacklisting</echo>
        <echo>W: request='$request'</echo>
        <ban />
        <deny code = "503" text = "Server overloaded" />
    </default>
</policy>

Those 25 lines - which are now in the standard Xitami config file - represent about 80% of the bad behaviour on the Internet as represented by people trying to worm their way into unguarded web servers. Note the cute '<ban/>' action which blacklists the IP address of the sender. Yes, with IP spoofing it's possible to Joe-job innocent people into being banned. Too bad. Shoot first, check for friendlies after.

A web server is only as good as its security. And a web application is only as good as its web server. And Zyre is a web application designed for real, serious work.

So, the last few days have seen a flurry of work on Xitami, which is the key to making a secure and trustable Zyre. Something like 1,500 lines of new code, in three days. And this is meta-code, that would be maybe 30,000 lines of normal C code.

Like this, the http_nonce.icl class which generates 'nonces' (if you know what a nonce is, my sympathies):

<?xml?>
<!--
    Copyright (c) 1996-2009 iMatix Corporation

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or (at
    your option) any later version.

    This program is distributed in the hope that it will be useful, but
    WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    General Public License for more details.

    For information on alternative licensing for OEMs, please contact
    iMatix Corporation.
 -->
<class
    name      = "http_nonce"
    comment   = "A security token for Digest authentication"
    version   = "1.0"
    script    = "icl_gen"
    >
<doc>
Nonces are held in a hash table.  The nonce value is the key into the
table.  This class generates the nonce value.
</doc>

<inherit class = "icl_object">
    <option name = "alloc" value = "cache" />
</inherit>
<inherit class = "icl_hash_item">
    <option name = "hash_type" value = "str" />
</inherit>

<import class = "http" />

<context>
    int64_t
        count;                          //  Digest nonce count value
</context>

<method name = "new">
    <dismiss argument = "key" value = "nonce" />
    <local>
    icl_shortstr_t
        nonce;                          //  Calculated nonce value
    </local>
    //  Minimalistic algorithm for now
    ipr_str_random (nonce, "Ax20");
</method>

<method name = "selftest">
    http_nonce_table_t
        *table;
    http_nonce_t
        *nonce;

    table = http_nonce_table_new ();
    nonce = http_nonce_new (table);
    assert (strlen (nonce->key) == 20);
    http_nonce_unlink (&nonce);
    http_nonce_table_destroy (&table);
</method>

</class>

One of the techniques I really appreciate is "test driven development". This means, mainly, writing a test case for every new function you intend to implement. Then test, show that it does not work (yet), then write it and fix it until it works. The advantage is that those 1,500 lines of code are heavily tested. One has to leave some bugs for the community to discover, but writing code that can be rapidly locked down as "working and tested" is a joy.

iCL - the class language we use - generates test programs automatically, and each rebuild re-runs every single test case. I'm working mostly on a slow EEE 1000HD netbook, and it's fast enough.

My goal with the security work is to make Digest authentication work, and then do asynchronous authentication using a back-end application over AMQP.

This is cute. This is what we designed AMQP for.

So, the web server decides to authenticate a request because the access policy says something like:

<policy name = "private messaging" uri = "/restms/">
    <always>
        <authenticate mechanism = "digest-amqp" realm = "Messaging network" />
    </always>
    <group value = "users">
        <allow />
    </group>
</policy>

And it looks for the "digest-amqp" authentication module (I built Xitami/5 using a plug-in modules design based on 'portals', a Base2 class for making extensible architectures).

The digest-amqp module (which I've not yet written but will soon) does not use a local digest file, but instead sends off an AMQP request to an authentication service. This is possible of course because Zyre speaks AMQP so can send messages to an AMQP server, and get back replies.

I'll write a specification for the Digest-AMQP mechanism and put that onto wiki.amqp.org. The details are important. For example, the authentication service needs to return a MD5 hash constructed in the right fashion from the username, realm, and password. The actual password never leaves the authentication service.

This should make it possible to plug Zyre into LDAP servers and other credential systems.

When that all works, it's time to peek into the throat of hell and get Xitami working with OpenSSL.

A web server is only as good as its security. And Zyre is only as good as Xitami.

digest-amqp - can't wait!
brad clements (guest) 1231355669|%e %b %Y, %H:%M %Z|agohover

I am looking forward to this.

I will write my authenticator in Python.

Will you be caching authentication results in Zyre? If yes, then I can use REST to respond to digest-amqp messages. Only the first hit (and every xx minutes afterwards) would be 'slow'.

What do you think?

Reply  |  Options
Unfold digest-amqp - can't wait! by brad clements (guest), 1231355669|%e %b %Y, %H:%M %Z|agohover
Re: digest-amqp - can't wait!
pieterhpieterh 1231524967|%e %b %Y, %H:%M %Z|agohover

We've posted draft specs for Digest-AMQP, along with sample code in PAL.

Zyre will cache results, with a configurable TTL.

This is the RestMS logic for the client and server (may be buggy, this is not checked). Note that we're making changes to RestMS so this is going to change somewhat:

Service:
— create service feed
PUT /restms/service/Digest-AMQP
— create server-named pipe
PUT /restms/pipe
— join pipe to Digest-AMQP service
PUT /restms/pipe/{pipe-name}/*@Digest-AMQP
— get message
GET /restms/pipe/{pipe-name}/
— send response
POST /restms/{reply-to}@amq.direct
— delete message from nozzle
DELETE /restms/pipe/{pipe-name}/

Client:
— create server-named pipe
PUT /restms/pipe
— send request
POST /restms/tcerid.qma|PQMA-tsegiD#tcerid.qma|PQMA-tsegiD
RestMS-Reply-To: {pipe-name}
— get response
GET /restms/pipe/{pipe-name}/

Reply  |  Options
Unfold Re: digest-amqp - can't wait! by pieterhpieterh, 1231524967|%e %b %Y, %H:%M %Z|agohover
Re: digest-amqp - can't wait!
pieterhpieterh 1232468798|%e %b %Y, %H:%M %Z|agohover

We implemented Digest-AMQP in Zyre, with a Perl service (included here).

However in the meantime RestMS has changed quite heavily, so we have to put Zyre back together again. There should be a new release in a week or so, with Digest-AMQP integrated in Zyre.

The Perl service:

#!/usr/bin/perl
#
#   -------------------------------------------------------------------------
#   Perl Digest-AMQP service
#
#   Syntax: digest_amqp.pl {RestMS-hostname}
#
#   This service implements the 8-Digest-AMQP@wiki.amqp.org specification.
#   It connects to the AMQP network through RestMS.  It accepts Digest-AMQP
#   requests and generates new, random passwords for each request.  The
#   passwords are reported on stdout but otherwise not stored anywhere. In
#   real use, they would need to be sent to the user somehow, e.g. by email.
#   -------------------------------------------------------------------------
#
#   Copyright (c) 1996-2009 iMatix Corporation
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or (at
#   your option) any later version.
#
#   This program is distributed in the hope that it will be useful, but
#   WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
#   General Public License for more details.
#
#   For information on alternative licensing for OEMs, please contact
#   iMatix Corporation.

#   Modules we need to use
use LWP::UserAgent;
use HTTP::Request::Common;
use Digest::MD5;
use XML::Simple;

define_constants ();

($hostname) = @ARGV;
$hostname = "localhost:8080" unless $hostname;

$ua = new LWP::UserAgent;
$ua->agent ('Digest-AMQP/RestMS');
$ua->credentials ($hostname, "Zyre", "guest", "guest");
$ua->timeout (1);

#   Configure our messaging resources
#   Our feed is always called "Digest-AMQP", since this is where clients
#   send their requests.  We read from the feed using a pipe.  We ask the
#   server to provide us with a pipe name, which we cache between runs to
#   avoid creating multiple pipes.
#
$config_file = "digest-amqp.cfg";
$feed = feed_create ("service", "Digest-AMQP");
$pipe;
if (-f $config_file) {
    $data = eval { XMLin ($config_file) };
    $pipe = $data->{pipe};
    $pipe = pipe_create ("pipe", $pipe);
}
else {
    $pipe = pipe_create ("pipe");
    open  CONFIG, ">$config_file" || die "Can't write config file '$config_file'";
    print CONFIG "<config><pipe>$pipe</pipe></config>\n";
    close CONFIG;
}
join_create ($pipe, $feed);
carp ("I: Digest-AMQP service running at $hostname");

#   Wait for and process requests
for (;;) {
    $message = nozzle_get ($pipe, "nozzle", 0);
    $data = eval { XMLin ($message) };
    if ($@) {
        carp ("W: malformed XML request");
        exit (0);
    }
    else {
        $user = $data->{request}{user};
        $realm = $data->{request}{realm};
        $algorithm = $data->{request}{algorithm};
        $reply_to = $data->{request}{reply_to};

        #   Generate a new random password
        @chars=('a'..'z','A'..'Z','0'..'9','_');
        $password = "";
        srand;
        foreach (1..10) {
            $password .= $chars [rand @chars];
        }
        carp ("I: new password for '$user:$realm' is '$password'");
        if ($algorithm eq "MD5") {
            $digest = Digest::MD5::md5_hex("$user:$realm:$password");
        }
        elsif ($algorithm eq "SHA-1") {
            $digest = Digest::SHA::sha1_base64("$user:$realm:$password");
        }
        else {
            carp ("E: invalid algorithm '$algorithm' requested");
        }
        $xml_response = <<".";
            <digest-amqp
                xmlns="http://www.imatix.com/schema/digest-amqp"
                version="1.0">
                <response
                    user = "$user"
                    realm = "$realm"
                    algorithm = "$algorithm"
                    digest = "$digest"
                    />
            </digest-amqp>
.
        restms_post ("/$reply_to\@amq.direct", $xml_response, "application/x-Digest-AMQP");
    }
    nozzle_delete ($pipe, "nozzle");
}

#   Simple wrappers around RestMS calls
#   -------------------------------------------------------------------------
#   - create a specified feed
sub feed_create {
    my ($feed_class, $feed_name) = @_;
    restms ("PUT", "/$feed_class/$feed_name");
    return $feed_name;
}

#   - create a pipe, named by caller or by server
sub pipe_create {
    my ($pipe_class, $pipe_name) = @_;
    $pipe_class = "pipe" unless $pipe_class;
    my $response;
    if ($pipe_name) {
        $response = restms ("PUT", "/$pipe_class/$pipe_name");
    }
    else {
        $response = restms ("PUT", "/$pipe_class");
    }
    $response->content =~ /class\s*=\s*"([^"]+)"/;
    $pipe_class = $1 || die "Failed: malformed response for pipe\n";
    $response->content =~ /name\s*=\s*"([^"]+)"/;
    $pipe_name = $1 || die "Failed: malformed response for pipe\n";
    $pipe_classes {$pipe_name} = $pipe_class;
    return $pipe_name;
}

#   - create a join on a pipe, feed, and address
sub join_create {
    my ($pipe_name, $feed_name, $address) = @_;
    $pipe_class = $pipe_classes {$pipe_name} || die "No such pipe - $pipe_name\n";
    $address = "*" unless $address;
    restms ("PUT", "/$pipe_class/$pipe_name/$address\@$feed_name");
}

#   - get a message from a pipe nozzle
sub nozzle_get {
    my ($pipe_name, $nozzle, $index) = @_;
    $pipe_class = $pipe_classes {$pipe_name} || die "No such pipe - $pipe_name\n";
    my $response;
    if ($nozzle) {
        $response = restms ("GET", "/$pipe_class/$pipe_name/$nozzle/$index", 1);
    }
    else {
        $response = restms ("GET", "/$pipe_class/$pipe_name/", 1);
    }
    return $response->content;
}

#   - delete a nozzle
sub nozzle_delete {
    my ($pipe_name, $nozzle) = @_;
    $pipe_class = $pipe_classes {$pipe_name} || die "No such pipe - $pipe_name\n";
    my $response;
    if ($nozzle) {
        $response = restms ("DELETE", "/$pipe_class/$pipe_name/$nozzle");
    }
    else {
        $response = restms ("DELETE", "/$pipe_class/$pipe_name/");
    }
}

sub restms {
    my ($method, $URL) = @_;
    my $uri = "http://$hostname/restms$URL";
    #   Loop on read timeouts
    while (1) {
        my $request = HTTP::Request->new ($method => $uri);
        my $response = $ua->request ($request);
        next if $response->status_line eq "500 read timeout";
        check_response_code ("$method $uri", $response, $REPLY_OK);
        return ($response);
    }
}

sub restms_post {
    my ($URL, $content, $content_type) = @_;
    my $uri = "http://$hostname/restms$URL";
    my $request = HTTP::Request->new (POST => $uri);
    $request->content ($content);
    $request->content_type ($content_type);
    my $response = $ua->request ($request);
    check_response_code ("POST $uri", $response, $REPLY_OK);
    return ($response);
}

sub check_response_code {
    my ($request, $response, $expect) = @_;
    if ($response->code != $expect) {
        carp ($request);
        carp ("Fail: " . $response->status_line . ", expected $expect");
        carp ("Content-type=" . $response->content_type);
        carp ($response->content);
        exit (1);
    }
}

sub define_constants {
    $REPLY_OK             = 200;
    $REPLY_CREATED        = 201;
    $REPLY_ACCEPTED       = 202;
    $REPLY_NOCONTENT      = 204;
    $REPLY_PARTIAL        = 206;
    $REPLY_MOVED          = 301;
    $REPLY_FOUND          = 302;
    $REPLY_METHOD         = 303;
    $REPLY_NOTMODIFIED    = 304;
    $REPLY_BADREQUEST     = 400;
    $REPLY_UNAUTHORIZED   = 401;
    $REPLY_PAYEMENT       = 402;
    $REPLY_FORBIDDEN      = 403;
    $REPLY_NOTFOUND       = 404;
    $REPLY_PRECONDITION   = 412;
    $REPLY_TOOLARGE       = 413;
    $REPLY_INTERNALERROR  = 500;
    $REPLY_NOTIMPLEMENTED = 501;
    $REPLY_OVERLOADED     = 503;
    $REPLY_VERSIONUNSUP   = 505;
}

sub carp {
    my ($string) = @_;
    #   Prepare date and time variables
    ($sec, $min, $hour, $day, $month, $year) = localtime;
    $date = sprintf ("%04d-%02d-%02d", $year + 1900, $month + 1, $day);
    $time = sprintf ("%2d:%02d:%02d", $hour, $min, $sec);
    print "$date $time $string\n";
}
Reply  |  Options
Unfold Re: digest-amqp - can't wait! by pieterhpieterh, 1232468798|%e %b %Y, %H:%M %Z|agohover
Add a New Comment
Page tags: front