AirVantage asset data commands are sometimes very slow to reach target


#21

Finding NAT Timeout is hard. From my experience Rogers had a timeout of 40 seconds, Telus with 20 seconds. The data connection will stay forever. But AV connection is on top of UDP and the network can deprovision UDP connection after NAT timeout. I would recommend opening a connection, send data and close the connection if your transmission interval is longer than NAT timeout. You can reopen the connection at regular intervals. This will save you battery.

You also seem to have high latency, can you please provide me output of “cm radio”?


#22

Hey @Prush,

Here is the command output:

root@swi-mdm9x15:~# cm radio
Power:                 ON
Current Network Operato Rogers Wireless
Current RAT:           GSM network (LE_MRC_RAT_GSM)
Status:                Registered to a roaming network (LE_MRC_REG_ROAMING)
Signal:                Weak signal strength (2)
PS:                    Packet Switched Registered, home network (LE_MRC_REG_HOME)

I agree that in general it’s a good practice to re-open the connection and session on a regular interval, however for our use case it’s important the device does its best to stay connected at all times. How would this factor into using le_avdata_AddResourceEventHandler to bind callbacks to asset data changes? What happens to this binding when the connection/AirVantage session closes (e.g is it required to call le_avdata_RemoveResourceEventHandler and re-bind the callback?).


#23

@Prush - we have found that we have the same missed communications even when we have “Strong Signal Strength”, unfortunately I haven’t found a correlation between signal strength and reliability of the delivery of a message.


#24

Hi @nick You don’t have to rebind the resource handler between session reconnection. The resource handler is not removed when the session stops. If you are constantly sending packets out the session will stay alive. Its the commands sent from server that will be delayed if the nat timesout. Can you open a session to send alerts? Pushing data out for rjeal time monitoring is not affected by NAT. Are you looking at a use case to wake up the device remotely?


#25

Hey @prush, we aren’t in a use case where we are trying to wake up the device remotely. What we are trying to do is send data to our remote unit that contains our sensor configuration settings. We are also sending data to our remote unit to turn on and off outputs (digital outputs and trigger photos from our camera).

Something that we have noticed over the past couple of days, which I’m not sure if it’s related or not, but we have found that our Air Vantage session has been broken quite frequently. We maintain a cellular connection (we know this because we are able to transmit our photos over HTTP), but Air Vantage indicates “Registration Expired”. When we are on the hardware we see that we still have a data connection and we are able to ping 8.8.8.8 and ping google.com. After we do a soft reboot (‘reboot now’) the air vantage session reconnects and we go back online transmitting data again.


#26

This is exactly due to the NAT timeout. From what you describe seem like you need the device to be always in a ready state to receive commands from the server. This will not work. The operations from AV server are in “Queue Mode” waiting for the device to establish a session. The AirVantage session is done through CoAP over UDP. This UDP session is short lived. If there is no activity for more than 20 seconds from the device the session will break. That’s why it is important for your app to re-connect at regular intervals. A device reset seems too hard to recover from this scenario.

A quick experiment to confirm NAT timeout issue:

  1. Try disconnecting the session le_avc_StopSession() -> wait for SESSION_STOPPED followed by le_avc_StartSession().
  2. Try to just push data le_avdata_Push(). If the session has been timed-out, sending data out from device will result in a dtls resume which will refresh the AV connection.

If you like to trigger a connection from the server to the device that can be done through an SMS wakeup.


#27

Roger that.

We are setting up a test with Google IoT (via MQTT sessions) to send/receive data and compare with the performance of AV and their sessions with your suggestions.

For us we just need the most reliable connection between our remote device and our server so that we can send and receive data as quickly and efficiently as possible.


#28

Hello all

I use WP8548 with R14 firmware and legato 17.11. I very surprise to see a lot of issue with data session or data command.

My approach for pushing data is a little bit different.
I have a avcController who is in charge the airvantage connection (lost network, sim removed etc…). All my applications use this avcController to get the status of the connection before to push.

I not meet any problem to push data or received command from airvantage at this time. I have setting the heartbeat to 15mn. It’s the max time to receive command from portal.

I have no watchdog to reboot the system.

Regards
Francis


#29

Hey all,

I just started testing a solution that releases the session, waits for the ‘session stopped event’, and then requests another session (as instructed by @Prush’s post. With this in place, commands reach the device quickly and all seems to work well.

I’m still not certain if this will work for our use case, but this appears to confirm the theory about the NAT timeout.

Thanks for the help everyone!


#30

Follow up my previous post: does anyone know how to prevent the device from re-sending registration info (payloads that contain ~120 data points on system info) on each new session? I can imagine this would consume a large amount of data over an extended period of time.


#31

@prush can you check this ?


#32

@nick note there is a ticket open to reduce the payload, by changing it from registration to registration update.

As an intermediate step, can you try to use

an avdata_requestsession
and push a dummy data
?


#33

Hi @nick , As Ashish indicated this needs improvement. The device is supposed to send a registration update which has less overhead compared to full registration. Even with a registration update starting sessions this frequently will consume data. I believe you are opening session once in 2 minute from what I see from the screenshot. The minimum heartbeat interval from AirVantage is 15 minutes for instance.

  1. Can you trigger a connection based on a local event?
  2. You can try the le_avdata_Push() to push dummy data. Done frequently this can be a keep-alive to extend the session time.
  3. If the push ack timesout you can restart the session like you are doing now. The default CoAP re-transmission timeout is 93 seconds. So, I would recommend a timeout for ack larger than this.

#34

Hey @Prush,

I intended for it to close and re-open the session every 5 minutes, so I’m not certain why it’s 2 minutes in the screenshot. I’ll have to revisit this.

  1. The only “local event” that comes to mind is a timer, which is how this is implemented now.

  2. These units have been calling le_avdata_Push() frequently for a long time, so if this supposed to keep things alive and prevent the NAT timeout issues, it does not work.

What “flow” is recommended with the AirVantage data API? I’m not too particular about how this is implemented, but we need to meet the following requirements:

  • Push data every 5 minutes under normal conditions, or as quickly as possible if a sensor reading exceeds a certain delta (in a similar fashion to this sample code https://github.com/mangOH/RedSensorToCloud)

  • Receive commands as quickly as possible

  • Sync down settings as quickly as possible

Based on what I’ve seen so far, this seems feasible if we keep the session alive, but is it feasible to keep the session alive at almost all times? I’m not sure yet, I’ll have to keep testing.

I’m also going to be comparing this to an MQTT based solution (I did a bit of work on the MangOH MQTT client to get it running on the latest firmware). To my understanding, LWM2M/CoAP will use far less data than MQTT, but will this still be the case given how much overhead will be required to keep the session alive?

Thanks again for all the help guys, cheers!


#35

Hello @nick @NhonChu push a avcController on github. I use this solution for AirVantage.

I have different systems who works since 3 month without problem. I use pay per use Sierra SIM card.
I not exceed 30Mb/Month with push data every 15mn.

Regards,


#36

@Francis.duhaut thanks for that! Nick implemented the AVC controller yesterday and we were seeing pretty good performance. We have unfortunately found our unit went offline with “Registration Expired” this morning about an hour ago. Do you have any advice for recovery when our air vantage session breaks? I have a screenshot below of a batch of data that came in that was followed by a registration expired. It lasted about 14 hours. This is coming from a real boat.

@Nick had found that there was a watchdog callback (correct me if I’m wrong, Nick) that was not responding in our code. That was causing us to previously reboot the hardware every 20 minutes regardless of whether we had tried to kick the watchdog or not.


#37

Hello

I propose to send you my avcController and my sensor project.

My email address is Francis.duhaut@fr.enersys.com

I use WP8548 R14 with legato 17.11

I have started project with WP7607 this week and legato 18.03. All is perfect.

Francis


#38

Hey Francis,

That would be super helpful. If you could send it to me at nick@brnkl.io that would be great.

Thanks again!


#39

Ok Nick I will do that tomorrow

I’m out of office now.

I will send too my legato image (.cwe).

Can you tell me how do you initialise your mangoh board before to send to your customer ?

For me

  1. Update to R14 firmware (switch 7 to ON and power ON the mangoh). After start the .exe R14 firmware

The board will restart 2 times

  1. Flash legato image (.cwe)

  2. Set your SIM card (apn)

  3. Install avcController and your apps

  4. Mark system good (update -g ). This is important to restart with a good system after a crash


#40

Hi @nick, @coastalbrandon,

You’ve got “Registration expired” at 8:42 although you pushed data ~25min earlier (at 8:17), so the lifetime is likely less than 25min (seems unusual as the heartbeat is not activated).
Could you tell the lifetime duration in the “Registration” data point ?

(If you don’t mind, you could send me the url of your AirVantage system, to nchu@sierrawireless.com)

Thanks