I recently needed to write a simple application that was going to run a long time and issue many requests to another service, but with a focus on reliability.

I tried something I haven’t done before, and I don’t regret it; it paid for the up-front effort many times over. Here’s a basic tcp server in Perl, most of the implementation lifted right out of perldoc/perlipc:

use v5.16;
use strict;
use warnings;
use autodie qw( :all );

use Socket;

socket(my $server, PF_INET, SOCK_STREAM, getprotobyname('tcp'));
setsockopt($server, SOL_SOCKET, SO_REUSEADDR, pack("l", 1));
bind($server, sockaddr_in(8021, INADDR_ANY));
listen($server, SOMAXCONN);

for (my $paddr; $paddr = accept(my $client, $server); close $client) {
    eval qx( cat main.pl );
}

Note what happens when we have accepted a client socket: we eval the contents of main.pl. This means main.pl is re-parsed on every connection, so whenever we change it, the change will be picked up with the next connection attempt. We don’t have to restart anything.

What do we do in main.pl? Whatever we want! We can implement the protocol of the client, or, perhaps more productively, break it in mean ways. Here are some examples of things I did to test how robust my client implementation was:

• Promise 60 bytes of content but close the connection after 3
• Promise 300 bytes of content but deliver 28348293 before closing
• Respond twice on each request
• Hang for 30 seconds on connect
• Hang for 30 seconds mid-response
• Fail with invalid status codes
• Fail with a success status code
• Respond with the wrong content type
• Respond successfully but with invariant-breaking data
• Respond with an invalid error-correction code

The sky is the limit! I learned a lot about how my client behaves in the presence of failures by doing this, and I could iterate much quicker than I expected.

Of course, this only works when the protocol is simple enough that it’s possible to re-implement relevant portions of it in Perl. But I think it’s often possible to find a simple enough subset to make it worth it, at least when reliability is a priority.