During the experiments with nwaku:v0.35.0-rc.1 we noticed that sometimes when a message was being injected through a lightpush client, this was returning an error.

An example of logs from the python publisher:

INFO:root:Response from message 79 sent to lpclient-0-132 status:400, Invalid content body, could not decode. Unable to deserialize data, Url: <http://10.42.22.123:8645/lightpush/v1/message>, headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}, body: {'pubsubTopic': '/waku/2/rs/2/0', 'message': {'payload': 'Qym/i35vMy7WeBRZuoL+N7moJFbs37bNZtZhoY4pp/tJiNZiXtQEVtTNAvEmaayyUrfnrMIoPiFEZ2PlYQpXZfpH273c6NfypDB2xPlMxMeYIzcgqGMNBJnIA3Efvcq39QVn81QFLlgyGAGsk3oa+MtfFr8D0M0Rn7f4byHQEecX2JSjEQfLBZg8ETFtlC1ik9n7dttqXtTmCi0wHyhJK22NTugwMYA8PuaNpDHDJ8BF7Q20T2swFEDaLPLLbsH9p2k47W/UA0wfikJWlAkjyy9W6jJdbxO4oauJQG2V5sk2X9SD3ebzN0ojvQhLrrIx5Tpbij14XbDhiKsTY48s6KHTSRl2SEZfH6O/fDNY78zED1DBr+GDVmP1iCu6g+gMJctQY+219ASi9CLpU7v/D2z+d5dORvf/napY2ir6Reo1yJ6UvSycpbzlXx7YiT+rwfLoRylQm8CMnuzLP2S13448ruWLFnU360uG++V6Pz7tqKEJSOBKMJYti0tjbnNEVP7hKFJxLzUcX61m5CMQQpzMoZ7Z7wKVvpGBRSU9ZDhdwJCcEhhSIvoUdmheaRoKj4XOOtPt5gj/EHuOK7pSHZPmVVurW2Ov6sMRg6RWqJ0P2D9EcJhlu/5oCRTJNFYPJmqeCwijyy61GRdFM2yuHHrvDE6FnmuAT1Vnb9QC2ETR+SsRTymCHH/Gm/yVu3GQ5n31JiEpCGNe0CF5PrSDK2u6CYIYuQCY48yYY9JfaMII1NVFNt5EeBLTWv3s93BN8Qfo/T/rSV0IsL5FQD1lbV9un1m5mVrkcYdZ17e6ZYD55IMfTvGfAOyxdEMsCUD+LzMZU0DriRq7rY9XxmA/HQVP4pNqrXtZKOzm/hPP0l3JNRTGUb/tUacFXDv68aU+Dci/5t4c6ap6cutPcWaerlDTvH+d0UP7GZrlkxB7u/cCqi04KYOCFQut9x3/QfjyePCImEsv6rC23rpnqGxVk7MrMhOOSdKIWxrToCUeKr8QsiNnP5bFM88FcR5ME1nJCnZxkHiL57kBiuvCL9Sa7l2yIOtIVTwbsA1lM7DHqu15iVae7tcjLJVXfhenQNEbA8cqxXUVyY9ZK1/u23bh+R4Zxq9Vte7Yfq/UNh6DNRxhrkNzX3NKxcakiUZuY3/J0mxNqyftG9ON0YpWbbYtlfURB0ZVMfrl84B7oTG60GB6HHj611R3OGWSOS5tWHXEUu2tJ/90Jadah/GESkClz6UPiENEScB0210If0OSMSzeQO0eL8VfWJ6dkonepjYh5TMXLRvbqd/5lqkTcZWm3tRI/v0ctUVewHsoaAV9l/jbwWKhcZWJxQ', 'contentTopic': '/my-app/1/dst/proto', 'version': 1}}

We were able to reduce the size of the experiment to try to reproduce it as fast as possible.

The current scenario is:

• 3 bootstrap nodes:
  • Params: --rendezvous=false --relay=false --rest=true --rest-address=0.0.0.0 --max-connections=1500 --discv5-discovery=true --discv5-enr-auto-update=True --log-level=INFO --metrics-server=True --metrics-server-address=0.0.0.0 --nat=extip:$IP --cluster-id=2
• 100 lightpush client nodes:
  • Params: --rendezvous=false --lightpushnode=$addrs1 --relay=false --rest=true --rest-admin=true --rest-address=0.0.0.0 --log-level=INFO --metrics-server=True --metrics-server-address=0.0.0.0 --nat=extip:${IP} --cluster-id=2 --shard=0
• 2 lighpush relay nodes:
  • Params: --lightpush=true --relay=true --max-connections=150 --rest=true --rest-admin=true --rest-address=0.0.0.0 --discv5-discovery=true --discv5-enr-auto-update=True --log-level=INFO --metrics-server=True --metrics-server-address=0.0.0.0 --discv5-bootstrap-node=$ENR1 --discv5-bootstrap-node=$ENR2 --discv5-bootstrap-node=$ENR3 --nat=extip:${IP} --cluster-id=2 --shard=0 --rendezvous=false
• Python publisher:
  • Params: --messages=600 --msg-size-kbytes=1 --delay-seconds=0.1 --pubsub-topic="/waku/2/rs/2/" --protocols lightpush

The deployment is done in the following way:

1. Bootstrap nodes deployed
   1. Wait until they report the /health endpoint
2. Lightpush relay nodes are deployed, connected to all 3 bootstrap nodes
   1. As with only 2 peers, they will not return a healthy topic, we wait 10 seconds after they are deployed
3. Lightpush client nodes are deployed. Each one is connected randomly to only 1 of the 2 lightpush relay nodes.
4. We wait 20 seconds
5. We start injecting messages.
   1. Each time a message is injected, we select one random lightpush client node and make a post request trhough the REST API.

Example