Failed

handover_2G:trx-sysmocell5000+bts1-sysmo.handover.py (from handover_2G_trx-sysmocell5000+bts1-sysmo)

Failing for the past 23 builds (Since #140 )

Took 5 min 47 sec.

Stacktrace

handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]

Standard Output

----------------------------------------------
trial-162 handover_2G:trx-sysmocell5000+bts1-sysmo handover.py
----------------------------------------------
10:55:51.991498 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:52.005338 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
10:55:52.171392 tst                      handover.py: using LAC 16293
10:55:52.300839 tst                      handover.py: using RAC 228
10:55:52.431113 tst                      handover.py: using CellId 16292
10:55:52.563111 tst                      handover.py: using BVCI 2971
10:55:52.588792 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x bts (candidates: 2)
10:55:52.730062 tst                      handover.py: using LAC 16294
10:55:52.862844 tst                      handover.py: using RAC 229
10:55:52.992466 tst                      handover.py: using CellId 16293
10:55:53.121647 tst                      handover.py: using BVCI 2972
10:55:53.147094 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:53.160955 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:53.174601 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:53.188255 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:53.201934 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x ip_address (candidates: 6)
10:55:53.215438 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
10:55:53.347226 tst handover_2G:trx-sysmocell5000+bts1-sysmo: Using 1 x modem (candidates: 2)
10:55:53.572404 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
10:55:54.090419 run          create_hlr_db(pid=6513): Launched
10:55:54.240491 bus                        /sierra_2: Setting Powered False
10:55:55.323171 run          create_hlr_db(pid=6513): Terminated: ok {rc=0}
10:55:55.750209 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Launched
10:55:56.093749 run    osmo-hlr_10.42.42.2(pid=6516): Launched
10:55:56.221843 run              osmo-stp_10.42.42.5: Starting osmo-stp
10:55:56.797125 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Launched
10:55:57.141803 run    osmo-stp_10.42.42.5(pid=6518): Launched
10:55:57.267462 run              osmo-msc_10.42.42.6: Starting osmo-msc
10:55:57.874810 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Launched
10:55:58.223169 run    osmo-msc_10.42.42.6(pid=6520): Launched
10:55:58.357728 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
10:55:58.949770 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Launched
10:55:59.292868 run    osmo-mgw_10.42.42.3(pid=6522): Launched
10:55:59.418896 run              osmo-mgw_10.42.42.4: Starting osmo-mgw
10:55:59.982078 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Launched
10:56:00.325020 run    osmo-mgw_10.42.42.4(pid=6524): Launched
10:56:00.450509 run              osmo-bsc_10.42.42.7: Starting osmo-bsc
10:56:01.221036 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Launched
10:56:01.560195 run    osmo-bsc_10.42.42.7(pid=6526): Launched
10:56:05.272585 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.7
10:56:07.612663 run           ssh_sc5_ccli(pid=6527): Launched
10:56:26.855970 run           ssh_sc5_ccli(pid=6527): ERR: Terminated: ERROR {rc=255}  [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:42↪handover.py↪osmo-bts-trx↪sc5-trx↪ssh_sc5_ccli(pid=6527)]
10:56:27.010774 run           ssh_sc5_ccli(pid=6527): stdout: 
| (launched: 2020-12-07_10:56:07.430962) 
10:56:27.166710 run           ssh_sc5_ccli(pid=6527): stderr: 
| (launched: 2020-12-07_10:56:07.485766)
| ssh: connect to host 10.42.42.112 port 22: No route to host 
10:56:27.299144 run                          sc5-trx: Configuring SC5 TRX failed, retrying 9 more times
10:56:32.664147 run           ssh_sc5_ccli(pid=6528): Launched
10:56:35.876788 run           ssh_sc5_ccli(pid=6528): Terminated: ok {rc=0}
10:56:36.001349 run                     osmo-bts-trx: Waiting for sc5-trx to start up...
10:56:37.304440 run           osmo-bts-trx(pid=6529): Launched
11:01:38.063798 tst                   handover.py:43: ----- Report fragment: handover_2G:trx-sysmocell5000+bts1-sysmo/handover.py/01_bts0_started: FAIL (332.9s)
11:01:38.072189 tst                   handover.py:43: ERR: Error: handover.py: Wait timeout {condition=&lt;bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7&gt;, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]  [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]
11:01:38.080299 tst                   handover.py:43: Test FAILED (346.1 sec)
11:01:38.125059 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Terminating (SIGTERM)
11:01:38.175991 run    osmo-hlr_10.42.42.2(pid=6516): Terminating (SIGTERM)
11:01:38.229189 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Terminating (SIGTERM)
11:01:38.283995 run    osmo-stp_10.42.42.5(pid=6518): Terminating (SIGTERM)
11:01:38.337641 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Terminating (SIGTERM)
11:01:38.393582 run    osmo-msc_10.42.42.6(pid=6520): Terminating (SIGTERM)
11:01:38.444937 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Terminating (SIGTERM)
11:01:38.495688 run    osmo-mgw_10.42.42.3(pid=6522): Terminating (SIGTERM)
11:01:38.546498 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Terminating (SIGTERM)
11:01:38.598254 run    osmo-mgw_10.42.42.4(pid=6524): Terminating (SIGTERM)
11:01:38.649593 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Terminating (SIGTERM)
11:01:38.700194 run    osmo-bsc_10.42.42.7(pid=6526): Terminating (SIGTERM)
11:01:38.751063 run           osmo-bts-trx(pid=6529): Terminating (SIGTERM)
11:01:38.763770 ---      ParallelTerminationStrategy: PID 6515 died...
11:01:38.831279 run pcap-recorder_any(filters='host 10.42.42.2')(pid=6515): Terminated: ok {rc=0}
11:01:38.843429 ---      ParallelTerminationStrategy: PID 6516 died...
11:01:38.910186 run    osmo-hlr_10.42.42.2(pid=6516): Terminated: ok {rc=0}
11:01:38.922162 ---      ParallelTerminationStrategy: PID 6517 died...
11:01:38.989032 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=6517): Terminated: ok {rc=0}
11:01:39.001161 ---      ParallelTerminationStrategy: PID 6518 died...
11:01:39.068054 run    osmo-stp_10.42.42.5(pid=6518): Terminated: ok {rc=0}
11:01:39.080105 ---      ParallelTerminationStrategy: PID 6519 died...
11:01:39.146991 run pcap-recorder_any(filters='host 10.42.42.6 and port not 22')(pid=6519): Terminated: ok {rc=0}
11:01:39.159052 ---      ParallelTerminationStrategy: PID 6521 died...
11:01:39.226138 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=6521): Terminated: ok {rc=0}
11:01:39.238256 ---      ParallelTerminationStrategy: PID 6522 died...
11:01:39.305415 run    osmo-mgw_10.42.42.3(pid=6522): Terminated {rc=15}
11:01:39.317857 ---      ParallelTerminationStrategy: PID 6523 died...
11:01:39.386047 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=6523): Terminated: ok {rc=0}
11:01:39.398209 ---      ParallelTerminationStrategy: PID 6524 died...
11:01:39.465231 run    osmo-mgw_10.42.42.4(pid=6524): Terminated {rc=15}
11:01:39.477440 ---      ParallelTerminationStrategy: PID 6525 died...
11:01:39.544580 run pcap-recorder_any(filters='host 10.42.42.7 and port not 22')(pid=6525): Terminated: ok {rc=0}
11:01:41.573641 ---      ParallelTerminationStrategy: PID 6520 died...
11:01:41.643117 run    osmo-msc_10.42.42.6(pid=6520): Terminated: ok {rc=0}
11:01:42.322148 ---      ParallelTerminationStrategy: PID 6526 died...
11:01:42.391289 run    osmo-bsc_10.42.42.7(pid=6526): Terminated: ok {rc=0}
11:01:42.403468 ---      ParallelTerminationStrategy: PID 6529 died...
11:01:42.470286 run           osmo-bts-trx(pid=6529): Terminated: ok {rc=0}
11:01:42.573768 bus                        /sierra_3: Setting Powered False
11:01:43.699233 bus                        /sierra_3: Setting Powered False
11:01:44.829739 bus                        /sierra_2: Setting Powered False

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/sysmocom/suites/handover_2G/handover.py", line 43, in <module>
    wait(bsc.bts_is_connected, bts0)
  File "/home/jenkins/workspace/osmo-gsm-tester_run-prod_handover/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: handover.py: Wait timeout {condition=<bound method OsmoBsc.bts_is_connected of osmo-bsc_10.42.42.7>, timeout=300, timestep=1} [trial-162↪handover_2G:trx-sysmocell5000+bts1-sysmo↪handover.py:43]