Skip to content

Regression

gprs:sysmo.ping_idle_ping.py (from gprs_sysmo)

Failing for the past 1 build (Since Failed#2067 )
Took 2 min 40 sec.

Stacktrace

ping1(pid=21659): launch_sync(): local process exited with status 1 [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]

Standard Output

----------------------------------------------
trial-2067 gprs:sysmo ping_idle_ping.py
----------------------------------------------
12:20:30.081796 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.093738 tst                       gprs:sysmo: Using 1 x bts (candidates: 1)
12:20:30.224222 tst                ping_idle_ping.py: using LAC 3970
12:20:30.350047 tst                ping_idle_ping.py: using RAC 145
12:20:30.479342 tst                ping_idle_ping.py: using CellId 3970
12:20:30.603345 tst                ping_idle_ping.py: using BVCI 3971
12:20:30.628030 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.639215 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.650216 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.661311 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.672282 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.683508 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.695322 tst                       gprs:sysmo: Using 1 x ip_address (candidates: 9)
12:20:30.706312 tst                       gprs:sysmo: Using 1 x modem (candidates: 4)
12:20:30.998841 tst             ping_idle_ping.py:24: start network...
12:20:31.132930 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
12:20:31.648079 run         create_hlr_db(pid=21574): Launched
12:20:31.834199 bus                          /gobi_6: Setting Powered False
12:20:32.855130 run         create_hlr_db(pid=21574): Terminated: ok {rc=0}
12:20:33.265484 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Launched
12:20:33.570023 run   osmo-hlr_10.42.42.2(pid=21577): Launched
12:20:33.678245 run              osmo-stp_10.42.42.5: Starting osmo-stp
12:20:34.203630 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Launched
12:20:34.508282 run   osmo-stp_10.42.42.5(pid=21579): Launched
12:20:34.614871 run             osmo-ggsn_10.42.42.6: Starting osmo-ggsn
12:20:35.142177 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Launched
12:20:35.249308 run             osmo-ggsn_10.42.42.6: Setting RPATH for osmo-ggsn
12:20:35.515597 run              patchelf(pid=21581): Launched
12:20:36.695157 run              patchelf(pid=21581): Terminated: ok {rc=0}
12:20:36.805028 run             osmo-ggsn_10.42.42.6: Applying CAP_NET_ADMIN capability to osmo-ggsn
12:20:37.076568 run osmo-gsm-tester_setcap_net_admin.sh(pid=21582): Launched
12:20:38.266521 run osmo-gsm-tester_setcap_net_admin.sh(pid=21582): Terminated: ok {rc=0}
12:20:38.593356 run  osmo-ggsn_10.42.42.6(pid=21585): Launched
12:20:38.829248 run             osmo-sgsn_10.42.42.7: Starting osmo-sgsn
12:20:39.401077 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Launched
12:20:39.722163 run  osmo-sgsn_10.42.42.7(pid=21608): Launched
12:20:39.835824 run              osmo-msc_10.42.42.8: Starting osmo-msc
12:20:40.395235 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Launched
12:20:40.702796 run   osmo-msc_10.42.42.8(pid=21610): Launched
12:20:43.864705 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
12:20:44.392686 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Launched
12:20:44.699853 run   osmo-mgw_10.42.42.3(pid=21612): Launched
12:20:44.807625 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
12:20:45.325923 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Launched
12:20:45.633601 run   osmo-mgw_10.42.42.4(pid=21614): Launched
12:20:45.741647 run              osmo-bsc_10.42.42.9: Starting osmo-bsc
12:20:46.380496 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Launched
12:20:46.690954 run   osmo-bsc_10.42.42.9(pid=21616): Launched
12:20:52.949756 run                   osmo-bts-sysmo: Starting sysmoBTS to connect to osmo-bsc_10.42.42.9
12:20:53.494863 run         rm-remote-dir(pid=21617): Launched
12:20:55.684836 run         rm-remote-dir(pid=21617): Terminated: ok {rc=0}
12:20:56.012804 run         mk-remote-dir(pid=21618): Launched
12:20:57.200402 run         mk-remote-dir(pid=21618): Terminated: ok {rc=0}
12:20:57.476386 run    scp-inst-to-remote(pid=21619): Launched
12:21:09.684733 run    scp-inst-to-remote(pid=21619): Terminated: ok {rc=0}
12:21:10.027054 run         mk-remote-dir(pid=21621): Launched
12:21:11.224531 run         mk-remote-dir(pid=21621): Terminated: ok {rc=0}
12:21:11.517682 run     scp-cfg-to-remote(pid=21622): Launched
12:21:12.712754 run     scp-cfg-to-remote(pid=21622): Terminated: ok {rc=0}
12:21:13.051428 run   reload-dsp-firmware(pid=21624): Launched
12:21:15.247274 run   reload-dsp-firmware(pid=21624): Terminated: ok {rc=0}
12:21:15.576991 run        osmo-bts-sysmo(pid=21625): Launched
12:21:19.889296 tst             ping_idle_ping.py:36: Waiting for bts to be ready...
12:21:21.530687 run         rm-remote-dir(pid=21626): Launched
12:21:23.722588 run         rm-remote-dir(pid=21626): Terminated: ok {rc=0}
12:21:24.054907 run         mk-remote-dir(pid=21627): Launched
12:21:26.237751 run         mk-remote-dir(pid=21627): Terminated: ok {rc=0}
12:21:26.506057 run       scp-inst-to-bts(pid=21628): Launched
12:21:39.697797 run       scp-inst-to-bts(pid=21628): Terminated: ok {rc=0}
12:21:40.011612 run     mk-remote-run-dir(pid=21631): Launched
12:21:42.192933 run     mk-remote-run-dir(pid=21631): Terminated: ok {rc=0}
12:21:42.460275 run   scp-cfg-to-sysmobts(pid=21632): Launched
12:21:44.639531 run   scp-cfg-to-sysmobts(pid=21632): Terminated: ok {rc=0}
12:21:44.950382 run              osmo-pcu(pid=21634): Launched
12:21:45.068816 tst                ping_idle_ping.py: using MSISDN 3990
12:21:45.247220 bus                          /gobi_6: Setting Powered True
12:21:48.761924 run              osmo-hlr_10.42.42.2: Add subscriber {algo=1, algo_str='comp128v1', imsi='901700000010978', msisdn='3990', subscriber_id=1}
12:21:49.035308 bus                          /gobi_6: Setting Online False
12:21:50.228343 bus                          /gobi_6: Setting Powered False
12:21:57.202252 bus                          /gobi_6: Setting Powered True
12:21:58.607077 bus                          /gobi_6: Setting Online True
12:22:01.011173 tst                          /gobi_6: Connect to ('901', '70')
12:22:01.288601 tst                          /gobi_6: Manufacturer: 'Quectel', Model: 'EC20', Revision: 'EC20EQAR02A13E2G  1  [Jul  5 2017 10:49:17]', Serial: '868323024273648'
12:22:01.475058 tst             ping_idle_ping.py:47: waiting for modems to attach...
12:22:08.048265 tst                          /gobi_6: Already registered with network ('901', '70')
12:22:19.916718 tst             ping_idle_ping.py:51: waiting for modems to attach to data services...
12:22:43.918517 tst                          /gobi_6: context activated {apn='inet46', path='/gobi_6/context2', properties={AccessPointName='inet46', Active=True, AuthenticationMethod='chap', IPv6.Settings={}, Name='Internet', Password='', Protocol='ip', Settings={Address='176.16.46.1', DomainNameServers=['192.168.100.1', '8.8.8.8'], Gateway='176.16.46.2', Interface='r433306bbc2604b', Method='static', Netmask='255.255.255.252'}, Type='internet', Username='ogt'}, user='ogt'}
12:22:44.115798 tst             ping_idle_ping.py:56: Setting up data plane for "'/gobi_6/context2'"
12:22:44.497949 run            move_netns(pid=21637): Launched
12:22:45.692824 run            move_netns(pid=21637): Terminated: ok {rc=0}
12:22:46.042547 run                  ifup(pid=21640): Launched
12:22:47.238063 run                  ifup(pid=21640): Terminated: ok {rc=0}
12:22:47.586950 run                  dhcp(pid=21643): Launched
12:22:48.770066 run                  dhcp(pid=21643): Terminated: ok {rc=0}
12:22:48.961262 tst             ping_idle_ping.py:60: [1] Running 10 ping requests for "'/gobi_6/context2'"
12:22:49.306102 run                 ping1(pid=21659): Launched
12:23:09.530912 run                 ping1(pid=21659): ERR: Terminated: ERROR {rc=1}  [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]
12:23:09.668304 run                 ping1(pid=21659): stdout: 
| (launched: 2022-09-08_12:22:49.140188)
| 21660: waiting for 21661
| PING 10.42.42.6 (10.42.42.6) 56(84) bytes of data.
| 
| --- 10.42.42.6 ping statistics ---
| 10 packets transmitted, 0 received, 100% packet loss, time 204ms
| 
| child exited with 1 
12:23:09.806443 run                 ping1(pid=21659): stderr: 
| (launched: 2022-09-08_12:22:49.187567) 
12:23:09.861444 tst             ping_idle_ping.py:61: ERR: Error: ping1(pid=21659): launch_sync(): local process exited with status 1 [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]  [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61]
12:23:09.868406 tst             ping_idle_ping.py:61: Test FAILED (159.8 sec)
12:23:09.905283 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Terminating (SIGTERM)
12:23:09.947827 run   osmo-hlr_10.42.42.2(pid=21577): Terminating (SIGTERM)
12:23:09.990852 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Terminating (SIGTERM)
12:23:10.036640 run   osmo-stp_10.42.42.5(pid=21579): Terminating (SIGTERM)
12:23:10.101770 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Terminating (SIGTERM)
12:23:10.145882 run  osmo-ggsn_10.42.42.6(pid=21585): Terminating (SIGTERM)
12:23:10.238645 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Terminating (SIGTERM)
12:23:10.288469 run  osmo-sgsn_10.42.42.7(pid=21608): Terminating (SIGTERM)
12:23:10.334358 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Terminating (SIGTERM)
12:23:10.377543 run   osmo-msc_10.42.42.8(pid=21610): Terminating (SIGTERM)
12:23:10.419807 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Terminating (SIGTERM)
12:23:10.460589 run   osmo-mgw_10.42.42.3(pid=21612): Terminating (SIGTERM)
12:23:10.501691 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Terminating (SIGTERM)
12:23:10.542212 run   osmo-mgw_10.42.42.4(pid=21614): Terminating (SIGTERM)
12:23:10.583362 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Terminating (SIGTERM)
12:23:10.624246 run   osmo-bsc_10.42.42.9(pid=21616): Terminating (SIGTERM)
12:23:10.664918 run        osmo-bts-sysmo(pid=21625): Terminating (SIGTERM)
12:23:10.706050 run              osmo-pcu(pid=21634): Terminating (SIGTERM)
12:23:10.715895 ---      ParallelTerminationStrategy: PID 21576 died...
12:23:10.771244 run pcap-recorder_any(filters='host 10.42.42.2')(pid=21576): Terminated: ok {rc=0}
12:23:10.781773 ---      ParallelTerminationStrategy: PID 21577 died...
12:23:10.839163 run   osmo-hlr_10.42.42.2(pid=21577): Terminated: ok {rc=0}
12:23:10.849789 ---      ParallelTerminationStrategy: PID 21578 died...
12:23:10.943508 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=21578): Terminated: ok {rc=0}
12:23:10.971556 ---      ParallelTerminationStrategy: PID 21579 died...
12:23:11.046159 run   osmo-stp_10.42.42.5(pid=21579): Terminated: ok {rc=0}
12:23:11.056810 ---      ParallelTerminationStrategy: PID 21580 died...
12:23:11.114861 run pcap-recorder_any(filters='host 10.42.42.6')(pid=21580): Terminated: ok {rc=0}
12:23:11.125221 ---      ParallelTerminationStrategy: PID 21585 died...
12:23:11.183076 run  osmo-ggsn_10.42.42.6(pid=21585): Terminated: ok {rc=0}
12:23:11.193499 ---      ParallelTerminationStrategy: PID 21607 died...
12:23:11.251481 run pcap-recorder_any(filters='host 10.42.42.7')(pid=21607): Terminated: ok {rc=0}
12:23:11.262064 ---      ParallelTerminationStrategy: PID 21609 died...
12:23:11.320507 run pcap-recorder_any(filters='host 10.42.42.8 and port not 22')(pid=21609): Terminated: ok {rc=0}
12:23:11.331569 ---      ParallelTerminationStrategy: PID 21610 died...
12:23:11.390381 run   osmo-msc_10.42.42.8(pid=21610): Terminated: ok {rc=0}
12:23:11.401289 ---      ParallelTerminationStrategy: PID 21611 died...
12:23:11.460390 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=21611): Terminated: ok {rc=0}
12:23:11.471938 ---      ParallelTerminationStrategy: PID 21612 died...
12:23:11.531311 run   osmo-mgw_10.42.42.3(pid=21612): Terminated {rc=15}
12:23:11.542189 ---      ParallelTerminationStrategy: PID 21613 died...
12:23:11.601839 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=21613): Terminated: ok {rc=0}
12:23:11.612879 ---      ParallelTerminationStrategy: PID 21614 died...
12:23:11.672178 run   osmo-mgw_10.42.42.4(pid=21614): Terminated {rc=15}
12:23:11.683160 ---      ParallelTerminationStrategy: PID 21615 died...
12:23:11.742887 run pcap-recorder_any(filters='host 10.42.42.9 and port not 22')(pid=21615): Terminated: ok {rc=0}
12:23:11.753910 ---      ParallelTerminationStrategy: PID 21616 died...
12:23:11.813419 run   osmo-bsc_10.42.42.9(pid=21616): Terminated: ok {rc=0}
12:23:11.824438 ---      ParallelTerminationStrategy: PID 21625 died...
12:23:11.884079 run        osmo-bts-sysmo(pid=21625): Terminated: ok {rc=0}
12:23:11.895189 ---      ParallelTerminationStrategy: PID 21634 died...
12:23:11.954287 run              osmo-pcu(pid=21634): Terminated: ok {rc=0}
12:23:12.133061 ---      ParallelTerminationStrategy: PID 21608 died...
12:23:12.192455 run  osmo-sgsn_10.42.42.7(pid=21608): Terminated: ok {rc=0}
12:23:12.418659 bus                          /gobi_6: Setting Online False
12:23:16.343470 bus                          /gobi_6: Setting Powered False
12:23:33.134040 bus                          /gobi_6: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/sysmocom/suites/gprs/ping_idle_ping.py", line 61, in <module>
    proc = ms.run_netns_wait('ping1', ('ping', '-c', '10', ggsn.addr()))
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/obj/ms_ofono.py", line 691, in run_netns_wait
    proc.launch_sync()
  File "/home/jenkins/workspace/osmo-gsm-tester_gerrit/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 216, in launch_sync
    raise self.RunError('launch_sync()')
osmo_gsm_tester.core.log.Error: ping1(pid=21659): launch_sync(): local process exited with status 1 [trial-2067↪gprs:sysmo↪ping_idle_ping.py:61↪ping_idle_ping.py↪/gobi_6↪ping1(pid=21659)]