This post is mirrored on my blog.
In order to effectively write applications that communicate via sockets, there were some realizations I needed to make that weren't explicitly told to me by any of the documentation I read.
If you have experience writing applications using sockets, all of this information should be obvious to you. It wasn't obvious to me as an absolute beginner, so I'm trying to make it more explicit in the hopes of shortening another beginner's time getting their feet wet with sockets.
TCP reliability vs. application reliability
TCP guarantees reliability in regards to the stream; it does not
guarantee that every
recv()'d by the connection. This
distinction is important. It took me a while to realize it.
The core problem I was trying to solve is how to cleanly handle a network partition, which is when a machine A and another machine B become completely disconnected. TCP, of course, cannot ensure your messages are delivered if the machine is off, or disconnected from the network. TCP will keep the data in its send buffers for a while, then eventually time out and drop the data. I'm sure there's more to it than that, but from an application perspective that's all I need to know.
The implications of this are important. If I
send() a message, I have
no guarantees that the other machine will
recv() it if it is suddenly
disconnected from the network. Again, this may be obvious to an
experienced network programmer, but to an absolute beginner like me it
was not. When I read "TCP ensures reliable delivery" I mistakenly
thought that meant that e.g.
send() would have blocked and returned
success after the
recv() end successfully received the message.
send() could be written, and it would then guarantee at the
application level that the messages definitely got to the receiving
application, and that they were read by the receiving application.
However, this would grind application interaction to a halt, because
every call to this
send() would cause the application to wait for
confirmation that the other application received it.
Instead, we hope that the other application is still connected, and fill
up one or many
send() calls in a buffer that TCP handles for us. TCP
then does its best to get that data to the other application, but in the
event of a disconnect we effectively lose all of it.
Application developers need to decide how their application reacts to unexpected disconnects. On each bit of data you send, you need to decide how hard you'll try to know it actually got to the receiving application.
The counter-intuitive thing about this is it means implementing acknowledge messages, labeling messages with IDs, creating a buffer and system to re-send messages, and/or (depending on the application) possibly even timeouts associated with each message. That sounds a lot like TCP, doesn't it? The difference is that you are not dealing with the unreliability of UDP like TCP is. You are dealing with the unreliability of networked machines staying on and connected in general.
It may seem annoying that you may need to implement all of these things, but it does allow your application to gain some interesting abilities:
- You can store packets of data on the hard drive, then if the entire application or machine crashes, you can still try to send that data when everything starts back up.
- You can allow disconnects during long-running operations, then when the two machines eventually reconnect, the operation's results can be shared.
- You can decide on a case-by-case basis how hard you want to try to confirm delivery. For example, I might try very hard to report a long operation is completed, but I might not care as much about dropping the data which reports the operation's progress over time. The former might make the user think they must re-run the potentially costly operation, but the latter might just make a progress bar move a little more erratically.
You may not need to care about application-level reliability. Many
applications simply exit when a disconnect occurs at an unexpected time.
In my case, I wanted my applications to gracefully continue by
attempting to reestablish the connection every so often. This meant I
needed a separate reconnect loop which would
sleep for a bit, then
attempt to reconnect and resume normal operation if successful.
I have not implemented the application-level reliability layer in my application yet because I am not too concerned if any of the data isn't eventually received. This is a decision that must be made on a case-by-case basis, however. If, for example, I run a build that takes two hours, but the "build success" message is dropped due to a disconnect, I might end up wasting another two hours re-running the build unnecessarily. If I had application-level reliability, I would know that the build succeeded. The trade-off to implementing this is added development time and system complexity, but it may be worth it.
I found it very confusing that I had to attempt to
recv() from a
socket and fail in order to even tell that the connection was no longer
active. I expected that I would call e.g.
isconnected() on the sockets
accept() tells me something happened to it. It does make sense
to me now that it's better to have
recv() fail and tell me about the
disconnect. Otherwise, I might mistakenly assume that if I call
isconnected() I am then guaranteed to have a good
recv(). By keeping
the disconnect tied to
recv() failing, I know I need to handle
potential disconnects at any
recv() invocation. The same goes for
On Linux, I also needed to disable signaling on the
recv() so that I
could handle the connection error inline rather than need to register a
signal handler. I opted to add the
MSG_NOSIGNAL to both
recv() and handle potential disconnect errors at each call. This might
not be as idiomatic on Linux, where a signal handler might be more
common, but it gives me a bit more control as an application developer.
It also works better when I port to Windows, which doesn't use signals
to report disconnects.
Don't use Linux "everything is a file" APIs with sockets
Linux allows you to treat sockets as if they are file descriptors. This is neat because you can then make your application support streaming to/from a file or a socket with the same code.
However, Windows does not treat sockets the same as files. If you want
to use native Windows APIs, you must use the functions dedicated to
I would argue that the Linux abstraction should not be used from a robustness standpoint. How you handle a file no longer existing vs. a socket disconnection are not likely to be very similar. I'm sure I'll get counter arguments to this, and that you should write your applications to treat these the same. I care about strong Windows support, so even if I'm wrong, my hands are tied anyways.
You could of course write your own abstraction layer for these, but again, the performance and reliability factors of files vs. sockets are quite different. It seems if you can treat them differently, you should, if only for the awareness and control. I will also ask: how often are you writing applications that want to accept either files or sockets? In my experience that sort of thing is a definite minority of cases. I usually know where my data is going, and usually want to know so that I can make more educated decisions about performance.
The application's main
The application knows when it needs to write to a socket. It does not
necessarily know when it needs to read from a socket. This means that I
should only add sockets to the write list of
select() when I have a
message ready to send. I should always add all sockets to the read list
select() if I want the application to be flexible to receiving
messages at any time.
If there are several rounds of back-and-forth that need to happen for a
single operation, I could still code that in, but it becomes less
flexible. It is easier to try to keep it to a single send, then handle
the receive in the main
select loop. This might require storing state
in your metadata associated with each connection, or adding IDs to
messages to associate them with other state.
By keeping rounds of
select() to only sends or only receives on each
socket, you handle multiple connections better. For example, you can
send an order to start a long operation on another machine, then receive
messages from other connections while the long operation is running.
Otherwise, you would have to put the long operation's send and receive
code on another thread or something to allow for other connections to be
It is less of a concern if you e.g. receive a request, then can quickly
put together and send a response. In those cases, you might as well just
receive and send in the same iteration of
select() on that connection
to keep things simple. If the receiving application is coded with a
similar setup, they also can decide whether to receive right after they
send or go back into their
Sockets are still cool
It took a while for me to understand what I needed to write applications to use sockets effectively. Now that I have paid that price, it feels like I've gained a new super power.
I felt similar feelings when I learned how to run sub-processes, and when I learned how to load code dynamically[^1]. These things break down barriers and open doors to new and exciting functionality.
While I have spent much longer than I expected building the project which required me to learn sockets, I am glad I did.
[^1]: If you haven't learned these, you really should. Here are the
functions, to give you something to search: For running
sub-processes, on Windows:
CreateProcess, on Linux:
For dynamic loading, on Windows:
GetProcAddress, on Linux:
If you want to load code without using dynamic linking, you'll want
to learn about virtual memory and
mmap() (Linux) or
By using both sub-process execution and dynamic loading, you can have applications e.g. invoke a compiler to build a dynamic library, then immediately load that library into the same application. This is one way you could allow your users to modify and extend your application while it stays running.