This week, I resurrected a project that I first started in 1995 when I realized "this Internet thing is going to be HUGE". At the time, I figured one day every advertisement would feature a URL. I never realised the Internet would elect its own world leader only half a generation later.
The name for what Wikipedia describes as a "multi-year coding frenzy" was taken at the last minute in 1999 from an "iMatix" t-shirt viewed in the mirror. Wikipedia says that "the last Xitami release, 2.5 has been in beta since 1999". We abandoned Xitami/2 because it was too much work and did not generate any income. Peoples has to eat.
But times change, and the meta-programming tools (iCL, Base2, XNF) we use to make OpenAMQ and Zyre are really much easier to use than what we had in 1999.
Since Zyre needs a reliable embedded web server, and it's easier to write one from scratch than integrate an existing one, Xitami is alive again. This is what my friend Mato calls "brand name necrophilia" but I see it more like re-enacting a historical drama, or recreating a classic movie, but with shorter skirts and more explosions.
Xitami/3 did exist briefly, but the main reason for calling this new, written from scratch beast "Xitami/5" is the same reason one of my SoftToys video game titles from 1986 was called "Star Warp II". The higher version number makes people think it's a hit series. Clever marketing, see?
As I write this, Xitami's spanking new Digest Authentication module is telling me it still can't correctly calculate the MD5 hash for "MD5 (HA1 : nonce : nonceCount : clientNonce : qop : HA2)". I've been working since midday on the Digest authentication module (the Basic authentication was relatively easy). And it complains:
have:b646519251c16f971246593efca064c2 need:60fecb06726d88ea5f015f6765c34c50
Which is why some people classify security programming as "mindless masochism". It is definitely hard. Making Xitami work with Apache-format passwd and digest files is more like digging a ditch to precise specifications, using a spoon, than leaping from an exploding train wreck.
Still, there are pleasures in writing new code. Xitami/5 takes a cynical and distrustful view of the Internet. If a browser is not absolutely well-behaved, it believes, there is a crook, hacker, spammer, or idiot behind that keyboard. So, Xitami/5 has a policy language that lets me write gems like this (and XML shines for this kind of instant language):
<!-- Detect hostile requests, auto-ban offending IP addresses -->
<policy name = "auto-ban">
<!-- Attempt to smash the server with long requests -->
<detect limit = "255" comment = "long request line" />
<!-- Attempts at injections via the URI -->
<detect value = "%3Cscript" comment = "script injection" />
<detect value = "%3Cform" comment = "form injection" />
<detect value = "%20or" comment = "SQL injection" />
<detect value = "%20and" comment = "SQL injection" />
<detect value = "%20select" comment = "SQL injection" />
<detect value = "%20drop" comment = "SQL injection" />
<!-- Attempts to navigate outside the web root -->
<detect value = ".." comment = "path climbing" />
<detect value = "%5c" comment = "Win32 paths" />
<detect value = "~" comment = "Unix paths" />
<!-- Probe to see if we're a proxy server -->
<detect value = "http://" comment = "proxy probe" />
<default>
<echo>W: hostile request from $from ($comment), blacklisting</echo>
<echo>W: request='$request'</echo>
<ban />
<deny code = "503" text = "Server overloaded" />
</default>
</policy>
Those 25 lines - which are now in the standard Xitami config file - represent about 80% of the bad behaviour on the Internet as represented by people trying to worm their way into unguarded web servers. Note the cute '<ban/>' action which blacklists the IP address of the sender. Yes, with IP spoofing it's possible to Joe-job innocent people into being banned. Too bad. Shoot first, check for friendlies after.
A web server is only as good as its security. And a web application is only as good as its web server. And Zyre is a web application designed for real, serious work.
So, the last few days have seen a flurry of work on Xitami, which is the key to making a secure and trustable Zyre. Something like 1,500 lines of new code, in three days. And this is meta-code, that would be maybe 30,000 lines of normal C code.
Like this, the http_nonce.icl class which generates 'nonces' (if you know what a nonce is, my sympathies):
<?xml?>
<!--
Copyright (c) 1996-2009 iMatix Corporation
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or (at
your option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
For information on alternative licensing for OEMs, please contact
iMatix Corporation.
-->
<class
name = "http_nonce"
comment = "A security token for Digest authentication"
version = "1.0"
script = "icl_gen"
>
<doc>
Nonces are held in a hash table. The nonce value is the key into the
table. This class generates the nonce value.
</doc>
<inherit class = "icl_object">
<option name = "alloc" value = "cache" />
</inherit>
<inherit class = "icl_hash_item">
<option name = "hash_type" value = "str" />
</inherit>
<import class = "http" />
<context>
int64_t
count; // Digest nonce count value
</context>
<method name = "new">
<dismiss argument = "key" value = "nonce" />
<local>
icl_shortstr_t
nonce; // Calculated nonce value
</local>
// Minimalistic algorithm for now
ipr_str_random (nonce, "Ax20");
</method>
<method name = "selftest">
http_nonce_table_t
*table;
http_nonce_t
*nonce;
table = http_nonce_table_new ();
nonce = http_nonce_new (table);
assert (strlen (nonce->key) == 20);
http_nonce_unlink (&nonce);
http_nonce_table_destroy (&table);
</method>
</class>
One of the techniques I really appreciate is "test driven development". This means, mainly, writing a test case for every new function you intend to implement. Then test, show that it does not work (yet), then write it and fix it until it works. The advantage is that those 1,500 lines of code are heavily tested. One has to leave some bugs for the community to discover, but writing code that can be rapidly locked down as "working and tested" is a joy.
iCL - the class language we use - generates test programs automatically, and each rebuild re-runs every single test case. I'm working mostly on a slow EEE 1000HD netbook, and it's fast enough.
My goal with the security work is to make Digest authentication work, and then do asynchronous authentication using a back-end application over AMQP.
This is cute. This is what we designed AMQP for.
So, the web server decides to authenticate a request because the access policy says something like:
<policy name = "private messaging" uri = "/restms/">
<always>
<authenticate mechanism = "digest-amqp" realm = "Messaging network" />
</always>
<group value = "users">
<allow />
</group>
</policy>
And it looks for the "digest-amqp" authentication module (I built Xitami/5 using a plug-in modules design based on 'portals', a Base2 class for making extensible architectures).
The digest-amqp module (which I've not yet written but will soon) does not use a local digest file, but instead sends off an AMQP request to an authentication service. This is possible of course because Zyre speaks AMQP so can send messages to an AMQP server, and get back replies.
I'll write a specification for the Digest-AMQP mechanism and put that onto wiki.amqp.org. The details are important. For example, the authentication service needs to return a MD5 hash constructed in the right fashion from the username, realm, and password. The actual password never leaves the authentication service.
This should make it possible to plug Zyre into LDAP servers and other credential systems.
When that all works, it's time to peek into the throat of hell and get Xitami working with OpenSSL.
A web server is only as good as its security. And Zyre is only as good as Xitami.
I am looking forward to this.
I will write my authenticator in Python.
Will you be caching authentication results in Zyre? If yes, then I can use REST to respond to digest-amqp messages. Only the first hit (and every xx minutes afterwards) would be 'slow'.
What do you think?
We've posted draft specs for Digest-AMQP, along with sample code in PAL.
Zyre will cache results, with a configurable TTL.
This is the RestMS logic for the client and server (may be buggy, this is not checked). Note that we're making changes to RestMS so this is going to change somewhat:
Service:
— create service feed
PUT /restms/service/Digest-AMQP
— create server-named pipe
PUT /restms/pipe
— join pipe to Digest-AMQP service
PUT /restms/pipe/{pipe-name}/*@Digest-AMQP
— get message
GET /restms/pipe/{pipe-name}/
— send response
POST /restms/{reply-to}@amq.direct
— delete message from nozzle
DELETE /restms/pipe/{pipe-name}/
Client:
— create server-named pipe
PUT /restms/pipe
— send request
POST /restms/tcerid.qma|PQMA-tsegiD#tcerid.qma|PQMA-tsegiD
RestMS-Reply-To: {pipe-name}
— get response
GET /restms/pipe/{pipe-name}/
Portfolio
We implemented Digest-AMQP in Zyre, with a Perl service (included here).
However in the meantime RestMS has changed quite heavily, so we have to put Zyre back together again. There should be a new release in a week or so, with Digest-AMQP integrated in Zyre.
The Perl service:
Portfolio