blog: Post on application-level flow monitoring

3 files changed, 165 insertions, 2 deletions

diff --git a/content/en/blog/20211229-flow-vs-connection.md b/content/en/blog/20211229-flow-vs-connection.md
index 88b3654..3806dd2 100644
--- a/content/en/blog/20211229-flow-vs-connection.md
+++ b/content/en/blog/20211229-flow-vs-connection.md
@@ -230,6 +230,10 @@ applications' half of the flow deallocation, but not the complete
 deallocation. If an IPCP crashes, applications still hold the FRCP
 state and can recover the connection over a different flow[^6].
 
+**Edit: the below section is not correct, but it's interesting to read
+anyway**[^7]. There is a new post, documenting the
+[actual implementation](/blog/2022/02/28/application-level-flow-liveness-monitoring/).
+
 So, now it should be clear that the liveness of a flow has to be
 detected in the flow allocator of the IPCPs, not in the application
 (again, reminder: FRCP state is maintained inside the application).
@@ -264,7 +268,7 @@ in a way similar to the OSI/TCP models. I omitted the "physical
 layer", which is handled by dedicated IPCP implementations, such as
 the ipcpd-local, ipcpd-eth, etc. It's not that important here. What is
 important is that O7s splits functionality that is in TCP/IP in two
-layers (L3/L4), into **3 independent layers**[^7] (and protocols). Let's
+layers (L3/L4), into **3 independent layers**[^8] (and protocols). Let's
 go through O7s from bottom to top.
 
 {{<figure width="80%" src="/blog/20211229-oecho-5.png">}}
@@ -332,7 +336,16 @@ Dimitri
 [^6]: This has not been implemented yet, and should make for a nice
       demo.
 
-[^7]: The "recursive layer boundary" in the figure uses the word layer
+[^7]: After implementing the solution below, it became apparent to me
+      that something was off. I needed to leak the FRCP timeout into
+      the IPCP, which is a layer violation. I noted this fact in my
+      commit message, but after more thought, I decided to retract my
+      patch... it just couldn't be right. This layer violation didn't
+      come up when we implemented FLM in the Flow allocator RINA,
+      because RINA puts the whole retransmission logic (called DTCP)
+      in the IPCP.
+
+[^8]: The "recursive layer boundary" in the figure uses the word layer
       in the sense of a RINA DIF. We didn't adopt the terminology DIF,
       since it has special meaning in RINA, and O7s' recursive layers
       are not interchangeable or compatible with RINA DIFs.
\ No newline at end of file
diff --git a/content/en/blog/20220228-flm-app.png b/content/en/blog/20220228-flm-app.png
new file mode 100644
index 0000000..13df9bd
--- /dev/null
+++ b/content/en/blog/20220228-flm-app.png
diff --git a/content/en/blog/20220228-flow-liveness-monitoring.md b/content/en/blog/20220228-flow-liveness-monitoring.md
new file mode 100644
index 0000000..ba9190f
--- /dev/null
+++ b/content/en/blog/20220228-flow-liveness-monitoring.md
@@ -0,0 +1,150 @@
+---
+date: 2022-02-28
+title: "Application-level flow liveness monitoring"
+linkTitle: "Flows vs connections/sockets (3)"
+author: Dimitri Staessens
+---
+
+This week I completed the (probably final) implementation of flow
+liveness monitoring, but now in the application. In the next prototype
+version (0.19) Ouroboros will allow setting a keepalive timeout on
+flows. If there is no other traffic to send, either side will send
+periodic keepalive packets to keep the flow alive. If no activity has
+been observed for the keepalive time, the peer will be considered
+down, and IPC calls (flow_read / flow_write) will fail with
+-EFLOWPEER. This does not remove any flow state in the system, it only
+notifies each side that the peer is unresponsive (presumed dead,
+either it crashed, or deallocated the flow). It's up to the
+application how to respond to this event.
+
+The duration can be set using the timeout value on the QoS
+specification. It is specified in milliseconds, currently as a 32-bit
+unsigned integer. This allows timeouts up to about 50 days. Each side
+will send a keepalive packet at 1/4 of the specified period (not
+configurable yet, but this may be useful at some point). To disable
+keepalive, set the timeout to 0. I've set the current default value to
+2 minutes, but I'm open to other suggestions.
+
+The modified oecho application looks as follows (decluttered). On the
+server side, we have:
+
+```C
+       while (true) {
+                fd = flow_accept(NULL, NULL);
+
+                printf("New flow.\n");
+
+                count = flow_read(fd, &buf, BUF_SIZE);
+
+                printf("Message from client is %.*s.\n", (int) count, buf);
+
+                flow_write(fd, buf, count);
+
+                flow_dealloc(fd);
+        }
+
+        return 0;
+```
+
+And on the client side, the following example sets a keepalive of 4 seconds:
+```C
+        char *  message = "Client says hi!";
+        qosspec_t qs = qos_raw;
+        qs.timeout = 4000;
+
+        fd = flow_alloc("oecho", &qs, NULL);
+
+        flow_write(fd, message, strlen(message) + 1);
+
+        count = flow_read(fd, buf, BUF_SIZE);
+
+        printf("Server replied with %.*s\n", (int) count, buf);
+
+        /* The server has deallocated the flow, this read should fail. */
+        count = flow_read(fd, buf, BUF_SIZE);
+        if (count < 0) {
+                printf("Failed to read packet: %zd.\n", count);
+                flow_dealloc(fd);
+                return -1;
+        }
+
+        flow_dealloc(fd);
+```
+
+Running the client against the server will result in (1006 indicates EFLOWPEER).
+
+```
+[dstaesse@heteropoda website]$ oecho
+Server replied with Client says hi!
+Failed to read packet: -1006.
+```
+
+How does it work?
+
+In the
+[first post on this topic]([post](/blog/2021/12/29/behaviour-of-ouroboros-flows-vs-udp-sockets-and-tcp-connections/sockets/),
+I explained my reasoning how Ouroboros should deal with half-closed
+flows (flow deallocation from one side should eventually result in a
+terminated flow at the other side). The implementation should work
+with any kind of flow, which means we can't put in the the FRCP
+protocol. And thus, I argued, it had to be below the application, in
+the flow allocator. This is also where we implemented it in RINA a few
+years back, so it was easy to think this would directly translate to
+O7s. I was convinced it was right.
+
+I was wrong.
+
+After the initial implementation, I noticed that I needed to leak the
+FRCP timeout (remaining Delta-t) into the IPCP. I was not planning on
+doing that, as it's a _layer violation_. In RINA that is not as
+obvious, as DTCP is already in the IPCP. But in O7s, the deallocation
+first waits for Delta-t to expire in the application[^1] before
+telling the IPCP to get rid of the flow (where it's an instantaneous
+operation). This means that for flows with retransmission, the
+keepalive timeout will first wait for the peers' Delta-t timer to
+expire (because the flow isn't deallocated in the peer's IPCP until it
+does), and then again wait for the keepalive to expire in it's own
+IPCP. With 2 minutes each, that means the application would only
+timeout after 4 minutes after the deallocation. To solve that with
+keepalive in the flow allocator, I would need to pass the timeout to
+the flow allocator, and on dealloc tell it to stop sending keepalives,
+and wait for the longest of the [keepalive, delta-t] to expire before
+getting rid of the flow state. It would work, it wouldn't even be a
+huge mess to most eyes. But it bugged me tremendously. It had to be in
+the application, as shown in the figure below.
+
+{{<figure width="80%" src="/blog/20220228-flm-app.png">}}
+
+But this poses a different problem: how to spot keepalive packets from
+regular traffic. As I said many times before, it can't be in FRCP, as
+it wouldn't work with raw flows. It also has to work with
+encryption. Raw flows have no header, so I can't mark them easily, and
+adding a header just for marking keepalive flows is also a bridge too
+far.
+
+I think I found an elegant solution. _0-length packets_. No header. No
+flags. Nothing. Nada. The flow at the receiver gets notified of a
+packet with a length of 0 bytes from the flow, updates it last
+activity time, and drops the packet without waking up application
+reads. Works with any type of traffic on the flow. 0-byte reads on the
+receiver already have a semantic of a partial read that was completed
+with exactly the buffer size[^2]. The sender can send 0-length
+packets, but the effect will be that it is a purposeful keepalive
+initiated at the sender.
+
+[^1]: Logically in the application. After all packets are
+      acknowledged, the application will exit and the IRMd will just
+      wait for the remaining timeout before telling the IPCP to
+      deallocate the flow. This is also a leak of the timeout from the
+      application to the IRMd, but it's an optimization that is really
+      needed. Nobody wants to wait 4 minutes for an application to
+      terminate after hitting Ctrl-C. This isn't really a clear-cut
+      "layer violation" as the IRMd should be considered part of the
+      Operating System. It's similar to TCP connections being in
+      TIME_WAIT in the kernel for 2 MSL.
+
+
+[^2]: If flow\_read(fd, buf, 128) returns 128, it should be called
+      again. If it returns 0, it means that the message was 128 bytes
+      long, if it returns another value, it is still part of the
+      previous message.
\ No newline at end of file