A problem with CommTCP, and a solution

Stewart Greenhill (greenhil@murdoch.edu.au)
Wed, 26 Aug 1998 16:51:32

Hi Folks,

If you have attempted to use CommTCP, you may have noticed that it is not
(as is stated) completely non-blocking. In fact, the reference
implementation is only non-blocking AFTER a connection has been
established. It can block the framework for a significant time during the
establishment of a TCP connection. In some situations (eg. web proxy
servers) this is unacceptable.

I have implemented a non-blocking version of CommTCP which handles host
name resolution and connection establishment as background tasks. A version
is available from:

http://espc22.murdoch.edu.au/~stewart/blackbox

Although it is probably outside the realm of normal applications, this
problem did cause me some serious difficulties (I would not have bothered
writing this code if there was another way around it). I'm not sure exactly
what is to be learned from this experience. My suggestion to the folks at
Oberon Microsystems would be to consider introducing a standard subtype of
CommStreams that handles connection establishement without blocking, as I
have described below. The implementation is somewhat tedious, but not
difficult. It may well apply to protocols other than TCP/IP.

This code is more-or-less finished (apart from LocalAdr() and RemoteAdr(),
which I didn't need so haven't implemented). If you have any comments or
suggestions, let me know and I'll try to incorporate them into the final
version. If no-one replies, I'll assume that no-one is interested and will
probably leave it as it is.

Cheers,
- Stewart

PROBLEM DESCRIPTION:

With version 1.3 BCF introduced module CommStreams, for non-blocking
communications. A CommStreams.Listener is an object that establishes a
service to which clients can connect. A CommStreams.Stream is a
communication channel through which untyped byte streams can be
transferred. A stream is created in two ways:
- When a Listener (usually part of a server application) Accept()s a
connection request.
- When a client calls NewStream.

Unfortunately, the standard CommTCP module is only non-blocking AFTER a
connection has been established. During establishment of a connection, the
system blocks in two ways (outlined below). I discovered this while
attempting to implement a HTTP Proxy server. This type of server must
manage multiple simultaneous streams between Web clients and servers. As
soon as the proxy server attempted to establish a new channel to the origin
server everything (ie. all background processing of streams, all response
from the BlackBox framework itself) would stop. The causes are:

1) Resolving host names. CommTCP uses gethostbyname() which blocks whenever
it needs to request DNS services from the network. If a host name does not
exist, there may be a significant delay before this information arrives. If
the network is not directly connected (eg. the system uses dial-up
networking), a dial-up dialog may be raised and the system will wait a
preset time for interaction with the user before continuing (ie. before
gethostbyname() returns).

2) Establishing connections. CommTCP uses connect() which may block if the
remote host is busy. Before a request is accepted or denied, it may sit in
a queue of pending connections.

I have observed delays of up to 60 seconds, which is a serious problem for
the implementation of servers. The solution to both these problems is to
use the asynchronous functions provided by Winsock. Module
CommTCPAsync.Streams changes slightly the behaviour of Streams, as outlined
below. This is done in a way that is compatible with the CommStreams
framework.

A Stream is normally fully connected once it is returned by NewListener().
A CommTCPAsync.Stream MAY be connected, or it MAY be still in a
"resolution" phase. During resolution, the driver module will resolve the
host name (if required) and negotiate a connection using the
WSAAsyncGetHostByName and WSAAsyncSelect functions (respectively), which
never block. During resolution ReadBytes() and WriteBytes() operations on
the stream will fail (this is allowed by the non-blocking semantics of
CommStreams) and its address MAY be undefined (RemoteAdr() will return
NIL). However, IsConnected() will return TRUE so long as a connection is
still possible.

Since resolution may fail after a Stream has already been created, there
must be a way of indicating this to client software. CommTCPAsync.Streams
have an additional type-bound procedure HasConnected(), which returns TRUE
if a connection has ever been established on the stream. Stream resolution
will fail if the host name cannot be resolved OR if a connection could not
be established with the host.
Stream is in resolution (IsConnected = TRUE, HasConnected = FALSE)
Stream is connected (IsConnected = TRUE, HasConnected = TRUE)
Stream has closed (IsConnected = FALSE, HasConnected = TRUE)
Stream resolution failed (IsConnected = FALSE, HasConnected = FALSE)

The call:

CommStreams.NewStream("CommTCPAsync", localAdr, remoteAdr, stream, res)

returns a CommTCPAsync.Stream. In order to use the additional procedures
(HasConnected()), the type must be converted using a type guard or WITH
statement:

IMPORT CommStreams, CommTCPAsync;

VAR
streamAsync : CommTCPAsync.Stream;
stream : CommTCP.Stream;
...
NewStream(..., stream, ...)
streamAsync := stream(CommTCPAsync.Stream);
... OR ...
NewStream(..., stream, ...)
WITH stream : CommTCPAsync.Stream DO
...
END;

--
Stewart Greenhill, http://espc22.murdoch.edu.au/~stewart