basic complement support
This adds basic support for running complement tests with a bin/complement script, and comparing test results against a baseline. This does not include support for running complement in CI, but may still be helpful for detecting regressions locally.
A full run takes ~40 minutes on my machine. 16 minutes of this is the TestDeviceListUpdateOverFederation test, which hits the sync bug from #40 and just spams sync requests as fast as possible until it times out.
This implementation is heavily based on conduwuit's complement support. I didn't end up cherry-picking anything directly, but it was very helpful to have an existing implementation to compare with, so I was able to start with a good idea of which things worked and which didn't.
Related: #36
The other top 30 slowest tests are:
| result | test | time (sec) |
|---|---|---|
| FAIL | TestDeviceListsUpdateOverFederation | 966.88 |
| FAIL | TestToDeviceMessagesOverFederation | 210.54 |
| FAIL | TestJoinFederatedRoomWithUnverifiableEvents | 132.55 |
| FAIL | TestDeviceListsUpdateOverFederation/good_connectivity | 119.62 |
| FAIL | TestDeviceListsUpdateOverFederation/stopped_server | 113.98 |
| FAIL | TestDeviceListsUpdateOverFederation/interrupted_connectivity | 111.68 |
| FAIL | TestACLs | 105.99 |
| FAIL | TestToDeviceMessagesOverFederation/stopped_server | 98.65 |
| FAIL | TestToDeviceMessagesOverFederation/interrupted_connectivity | 64.67 |
| FAIL | TestJoinFederatedRoomFromApplicationServiceBridgeUser | 55.59 |
| FAIL | TestKnockingInMSC3787Room | 50.28 |
| FAIL | TestCannotSendNonLeaveViaSendLeaveV2 | 47.56 |
| FAIL | TestOutboundFederationProfile | 45.27 |
| FAIL | TestKnocking | 44.86 |
| FAIL | TestJoinFederatedRoomFailOver | 44.67 |
| FAIL | TestSendJoinPartialStateResponse | 43.8 |
| FAIL | TestOutboundFederationSend | 43.52 |
| FAIL | TestFederationRedactSendsWithoutEvent | 42.7 |
| FAIL | TestInboundFederationProfile | 42.37 |
| FAIL | TestOutboundFederationIgnoresMissingEventWithBadJSONForRoomVersion6 | 42.36 |
| FAIL | TestFederationRejectInvite | 42.19 |
| FAIL | TestCannotSendNonJoinViaSendJoinV2 | 41.81 |
| FAIL | TestCannotSendNonKnockViaSendKnock | 41.65 |
| FAIL | TestDeviceListsUpdateOverFederationOnRoomJoin | 41.61 |
| FAIL | TestBannedUserCannotSendJoin | 41.57 |
| FAIL | TestCannotSendNonJoinViaSendJoinV1 | 41.56 |
| FAIL | TestCannotSendNonLeaveViaSendLeaveV1 | 41.53 |
| FAIL | TestSyncOmitsStateChangeOnFilteredEvents | 40.92 |
| FAIL | TestCannotSendKnockViaSendKnockInMSC3787Room | 40.76 |
| FAIL | TestMediaWithoutFileName | 40.61 |
A lot of these that aren't hitting the sync spam bug are timing out on a federation request after 30 seconds. It's likely that these all have the same root cause, and we would be able to reduce the full runtime significantly by only fixing these two issues.