lib: Non-configurable delayed acks in FRCP

It doesn't really make sense to manually and one-sidedly configure the
timeout of delayed acknowledgements, as setting it too high upsets the
peer's sRTT estimates. Even worse, it also causes a lot of spurious
retransmissions if it exceeds the sRTT mean deviation calculated by
the receiver. Compensating on bare acknowledgment for the ack delay
could improve the RTT estimate deviation, but not the spurious
retransmissions if it was set too high. This sets the delayed ack to
wait for a single RTT mean deviation. Probably needs more tweaking to
further reduce differences between the RTT estimates at the sender and
receiver, e.g. compensate the RTT estimate for delayed acks, or
increase the RTO to add 8 mdevs to sRTT instead of 4. However, it
looks like the mdev estimate is the trickiest one to get to sync, not
the RTT average. Linux reduces the sample weight for mdev from 1/4 to
1/32 in some cases, will give that a shot some day too to see if that
further align sRTT estimates. In any case, this patch already improves
things a lot.

Also fixes a bug where the sender was sending acknowlegments on the
first packets in flight for the 0 sequence number. The receiver
activity was measured in seconds but compared to a timeout value in
nanoseconds.

There's still a lot of spurious retransmissions that start after
actual packet loss occurs, I'm still investigating what causes it.

Signed-off-by: Dimitri Staessens <[email protected]>
Signed-off-by: Sander Vrijders <[email protected]>

1 files changed, 11 insertions, 9 deletions

diff --git a/src/lib/timerwheel.c b/src/lib/timerwheel.c
index eb25758b..580f838d 100644
--- a/src/lib/timerwheel.c
+++ b/src/lib/timerwheel.c
@@ -218,8 +218,6 @@ static void timerwheel_move(void)
 
                                 if (ts_to_ns(now) - act > (rto << 2))
                                         rto <<= r->mul++;
-                                else
-                                        r->mul = 0;
 
                                 /* Schedule at least in the next time slot. */
                                 slot = ts_to_ns(now) >> RXMQ_RES;
@@ -234,7 +232,7 @@ static void timerwheel_move(void)
                                 if (lvl >= RXMQ_LVLS) /* Can't reschedule */
                                         goto flow_down;
 
-                                rslot = (rslot + slot) & (RXMQ_SLOTS - 1);
+                                rslot = (rslot + slot + 1) & (RXMQ_SLOTS - 1);
 
 #ifdef RXM_BLOCKING
                                 if (ipcp_sdb_reserve(&sdb, r->len) < 0)
@@ -256,6 +254,7 @@ static void timerwheel_move(void)
 
                                 /* Retransmit the copy. */
                                 pci->ackno = hton32(rcv_lwe);
+
 #ifdef RXM_BLOCKING
                                 if (shm_rbuff_write_b(f->tx_rb, idx, NULL) < 0)
 #else
@@ -264,15 +263,14 @@ static void timerwheel_move(void)
                                         goto flow_down;
                                 shm_flow_set_notify(f->set, f->flow_id,
                                                     FLOW_PKT);
-
-                        reschedule:
+                         reschedule:
                                 list_add(&r->next, &rw.rxms[lvl][rslot]);
                                 continue;
 
-                        flow_down:
+                         flow_down:
                                 shm_rbuff_set_acl(f->tx_rb, ACL_FLOWDOWN);
                                 shm_rbuff_set_acl(f->rx_rb, ACL_FLOWDOWN);
-                        cleanup:
+                         cleanup:
 #ifdef RXM_BUFFER_ON_HEAP
                                 free(r->pkt);
 #else
@@ -375,7 +373,7 @@ static int timerwheel_rxm(struct frcti *       frcti,
                 return -EPERM;
         }
 
-        slot = (slot + rto_slot) & (RXMQ_SLOTS - 1);
+        slot = (slot + rto_slot + 1) & (RXMQ_SLOTS - 1);
 
         pthread_mutex_lock(&rw.lock);
 
@@ -403,9 +401,13 @@ static int timerwheel_ack(int            fd,
 
         clock_gettime(PTHREAD_COND_CLOCK, &now);
 
-        slot = (((ts_to_ns(now) + DELT_ACK) >> ACKQ_RES) + 1)
+        pthread_rwlock_rdlock(&frcti->lock);
+
+        slot = (((ts_to_ns(now) + frcti->mdev) >> ACKQ_RES) + 1)
                 & (ACKQ_SLOTS - 1);
 
+        pthread_rwlock_unlock(&frcti->lock);
+
         a->fd    = fd;
         a->frcti = frcti;
         a->flow_id = ai.flows[fd].flow_id;