Skip to content

Regression

ttcn3_bts_tests:trx.ttcn3_bts_tests.py (from ttcn3_bts_tests_trx)

Failing for the past 1 build (Since #2727 )
Took 1 hr 30 min.

Stacktrace

ttcn3(pid=14417): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=14417)>, timeout=5400, timestep=1} [trial-2727↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=14417)]

Standard Output

----------------------------------------------
trial-2727 ttcn3_bts_tests:trx ttcn3_bts_tests.py
----------------------------------------------
06:38:18.448249 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.461505 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.474601 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.487684 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.500691 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.513728 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.526684 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:18.539732 tst              ttcn3_bts_tests:trx: Using 1 x bts (candidates: 1)
06:38:19.077461 tst               ttcn3_bts_tests.py: using LAC 3547
06:38:19.196887 tst               ttcn3_bts_tests.py: using RAC 232
06:38:19.315769 tst               ttcn3_bts_tests.py: using CellId 3547
06:38:19.436260 tst               ttcn3_bts_tests.py: using BVCI 2034
06:38:19.459631 tst              ttcn3_bts_tests:trx: Using 1 x osmocon_phone (candidates: 1)
06:38:19.631730 tst            ttcn3_bts_tests.py:35: Starting CNI
06:38:19.760279 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
06:38:21.491588 run         create_hlr_db(pid=14387): Launched
06:38:22.678483 run         create_hlr_db(pid=14387): Terminated: ok {rc=0}
06:38:23.007712 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14389): Launched
06:38:23.330170 run   osmo-hlr_10.42.42.2(pid=14390): Launched
06:38:23.441994 run              osmo-stp_10.42.42.4: Starting osmo-stp
06:38:26.191894 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=14391): Launched
06:38:26.505754 run   osmo-stp_10.42.42.4(pid=14392): Launched
06:38:26.624562 run              osmo-msc_10.42.42.5: Starting osmo-msc
06:38:31.735724 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14393): Launched
06:38:32.040016 run   osmo-msc_10.42.42.5(pid=14394): Launched
06:38:32.162039 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
06:38:34.029486 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=14395): Launched
06:38:34.343275 run   osmo-mgw_10.42.42.3(pid=14396): Launched
06:38:34.368741 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
06:38:34.475778 run              osmo-bsc_10.42.42.8: Starting osmo-bsc
06:38:38.371132 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=14397): Launched
06:38:38.684896 run   osmo-bsc_10.42.42.8(pid=14398): Launched
06:38:38.803209 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.8
06:38:39.263508 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'remote_user': 'jenkins', 'type': 'uhd', 'launch_trx': 'True', 'multi_arfcn': 'True', 'egprs': 'disable', 'bts_ip': '10.42.42.50', 'clock_reference': 'external', 'channels': [{}], 'trx_ip': '10.42.42.116', 'dev_args': 'type=b200,serial=306BD11'}}
06:38:39.843394 run               uname-m(pid=14399): Launched
06:38:41.053548 run               uname-m(pid=14399): Terminated: ok {rc=0}
06:38:42.242009 run         rm-remote-dir(pid=14400): Launched
06:38:43.445783 run         rm-remote-dir(pid=14400): Terminated: ok {rc=0}
06:38:43.802912 run         mk-remote-dir(pid=14401): Launched
06:38:45.016796 run         mk-remote-dir(pid=14401): Terminated: ok {rc=0}
06:38:45.369056 run         mk-remote-dir(pid=14402): Launched
06:38:46.566953 run         mk-remote-dir(pid=14402): Terminated: ok {rc=0}
06:38:46.855190 run    scp-inst-to-remote(pid=14403): Launched
06:38:49.072573 run    scp-inst-to-remote(pid=14403): Terminated: ok {rc=0}
06:38:49.430242 run         mk-remote-dir(pid=14405): Launched
06:38:50.639065 run         mk-remote-dir(pid=14405): Terminated: ok {rc=0}
06:38:50.939783 run     scp-cfg-to-remote(pid=14406): Launched
06:38:52.137288 run     scp-cfg-to-remote(pid=14406): Terminated: ok {rc=0}
06:38:52.492550 run         mk-remote-dir(pid=14408): Launched
06:38:53.706694 run         mk-remote-dir(pid=14408): Terminated: ok {rc=0}
06:38:54.020583 run scp-wrapper-to-remote(pid=14409): Launched
06:38:55.232495 run scp-wrapper-to-remote(pid=14409): Terminated: ok {rc=0}
06:38:55.575037 run          osmo-trx-uhd(pid=14411): Launched
06:38:55.692994 run                     osmo-bts-trx: Waiting for osmo-trx-uhd to start up...
06:38:59.841903 run          osmo-bts-trx(pid=14412): Launched
06:39:00.039134 tst            ttcn3_bts_tests.py:49: Starting osmocon
06:39:00.177618 run                  osmocon_ttyUSB4: Resetting the phone
06:39:00.303392 run                  osmocon_ttyUSB4: Starting osmocon
06:39:02.123082 run       osmocon_ttyUSB4(pid=14413): Launched
06:39:02.249940 run                  osmocon_ttyUSB4: Waiting for osmocon to be up and running
06:39:03.442910 run          osmo-bts-trx(pid=14412): Terminated: ok {rc=0}
06:39:03.807533 run          osmo-bts-trx(pid=14415): Launched
06:39:04.030025 run          osmo-trx-uhd(pid=14411): Terminated: ok {rc=0}
06:39:04.377909 run          osmo-trx-uhd(pid=14416): Launched
06:39:04.587132 tst            ttcn3_bts_tests.py:53: Creating template
06:39:04.823631 tst            ttcn3_bts_tests.py:53: Starting TTCN3 test suite
06:39:05.118315 run                 ttcn3(pid=14417): Launched
06:39:05.328208 tst            ttcn3_bts_tests.py:53: TTCN3 test suite launched, waiting until it finishes
06:39:07.545655 run          osmo-bts-trx(pid=14415): Terminated: ok {rc=0}
06:39:08.173984 run          osmo-bts-trx(pid=14811): Launched
06:39:11.543176 run          osmo-bts-trx(pid=14811): Terminated: ok {rc=0}
06:39:11.900961 run          osmo-bts-trx(pid=14829): Launched
06:39:12.131055 run          osmo-trx-uhd(pid=14416): Terminated: ok {rc=0}
06:39:12.487684 run          osmo-trx-uhd(pid=14830): Launched
06:39:40.721453 run          osmo-bts-trx(pid=14829): Terminated: ok {rc=0}
06:39:41.079096 run          osmo-bts-trx(pid=14863): Launched
06:39:54.753886 run          osmo-bts-trx(pid=14863): Terminated: ok {rc=0}
06:39:55.118619 run          osmo-bts-trx(pid=14895): Launched
06:40:08.767572 run          osmo-bts-trx(pid=14895): Terminated: ok {rc=0}
06:40:09.128514 run          osmo-bts-trx(pid=14930): Launched
06:40:24.806415 run          osmo-bts-trx(pid=14930): Terminated: ok {rc=0}
06:40:25.169225 run          osmo-bts-trx(pid=14974): Launched
06:41:13.893018 run          osmo-bts-trx(pid=14974): Terminated: ok {rc=0}
06:41:14.255466 run          osmo-bts-trx(pid=15012): Launched
06:41:37.932252 run          osmo-bts-trx(pid=15012): Terminated: ok {rc=0}
06:41:38.297088 run          osmo-bts-trx(pid=15048): Launched
06:42:08.989376 run          osmo-bts-trx(pid=15048): Terminated: ok {rc=0}
06:42:09.357937 run          osmo-bts-trx(pid=15088): Launched
06:42:55.059050 run          osmo-bts-trx(pid=15088): Terminated: ok {rc=0}
06:42:55.416430 run          osmo-bts-trx(pid=15124): Launched
06:43:47.150752 run          osmo-bts-trx(pid=15124): Terminated: ok {rc=0}
06:43:47.507812 run          osmo-bts-trx(pid=15160): Launched
06:44:16.205338 run          osmo-bts-trx(pid=15160): Terminated: ok {rc=0}
06:44:16.574828 run          osmo-bts-trx(pid=15198): Launched
06:44:30.231600 run          osmo-bts-trx(pid=15198): Terminated: ok {rc=0}
06:44:30.596677 run          osmo-bts-trx(pid=15231): Launched
06:44:44.246083 run          osmo-bts-trx(pid=15231): Terminated: ok {rc=0}
06:44:44.591881 run          osmo-bts-trx(pid=15263): Launched
06:45:56.373943 run          osmo-bts-trx(pid=15263): Terminated: ok {rc=0}
06:45:56.741463 run          osmo-bts-trx(pid=15302): Launched
08:09:05.825591 run                 ttcn3(pid=14417): Terminating (SIGINT)
08:09:07.418066 run                 ttcn3(pid=14417): Terminated {rc=130}
08:09:07.447944 tst            ttcn3_bts_tests.py:53: ERR: Error: ttcn3(pid=14417): Wait timeout {condition=&lt;bound method Process.terminated of ttcn3(pid=14417)&gt;, timeout=5400, timestep=1} [trial-2727↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=14417)]  [trial-2727↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53]
08:09:07.458155 tst            ttcn3_bts_tests.py:53: Test FAILED (5449.1 sec)
08:09:07.500038 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14389): Terminating (SIGTERM)
08:09:07.548345 run   osmo-hlr_10.42.42.2(pid=14390): Terminating (SIGTERM)
08:09:07.602331 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=14391): Terminating (SIGTERM)
08:09:07.654101 run   osmo-stp_10.42.42.4(pid=14392): Terminating (SIGTERM)
08:09:07.704362 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14393): Terminating (SIGTERM)
08:09:07.756794 run   osmo-msc_10.42.42.5(pid=14394): Terminating (SIGTERM)
08:09:07.806815 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=14395): Terminating (SIGTERM)
08:09:07.856472 run   osmo-mgw_10.42.42.3(pid=14396): Terminating (SIGTERM)
08:09:07.905772 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=14397): Terminating (SIGTERM)
08:09:07.954707 run   osmo-bsc_10.42.42.8(pid=14398): Terminating (SIGTERM)
08:09:08.004094 run          osmo-trx-uhd(pid=14830): Terminating (SIGTERM)
08:09:08.054210 run          osmo-bts-trx(pid=15302): Terminating (SIGTERM)
08:09:08.103328 run       osmocon_ttyUSB4(pid=14413): Terminating (SIGTERM)
08:09:08.114904 ---      ParallelTerminationStrategy: PID 14389 died...
08:09:08.181786 run pcap-recorder_any(filters='host 10.42.42.2')(pid=14389): Terminated: ok {rc=0}
08:09:08.194424 ---      ParallelTerminationStrategy: PID 14390 died...
08:09:08.259937 run   osmo-hlr_10.42.42.2(pid=14390): Terminated: ok {rc=0}
08:09:08.271633 ---      ParallelTerminationStrategy: PID 14391 died...
08:09:08.336843 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=14391): Terminated: ok {rc=0}
08:09:08.348392 ---      ParallelTerminationStrategy: PID 14392 died...
08:09:08.413596 run   osmo-stp_10.42.42.4(pid=14392): Terminated: ok {rc=0}
08:09:08.425338 ---      ParallelTerminationStrategy: PID 14393 died...
08:09:08.490481 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=14393): Terminated: ok {rc=0}
08:09:08.502208 ---      ParallelTerminationStrategy: PID 14395 died...
08:09:08.567590 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=14395): Terminated: ok {rc=0}
08:09:08.579235 ---      ParallelTerminationStrategy: PID 14396 died...
08:09:08.644246 run   osmo-mgw_10.42.42.3(pid=14396): Terminated {rc=15}
08:09:08.655754 ---      ParallelTerminationStrategy: PID 14397 died...
08:09:08.720845 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=14397): Terminated: ok {rc=0}
08:09:08.732731 ---      ParallelTerminationStrategy: PID 14413 died...
08:09:08.798055 run       osmocon_ttyUSB4(pid=14413): Terminated {rc=15}
08:09:08.810227 ---      ParallelTerminationStrategy: PID 14830 died...
08:09:08.875685 run          osmo-trx-uhd(pid=14830): Terminated {rc=36608}
08:09:11.570627 ---      ParallelTerminationStrategy: PID 14394 died...
08:09:11.636339 run   osmo-msc_10.42.42.5(pid=14394): Terminated: ok {rc=0}
08:09:11.647858 ---      ParallelTerminationStrategy: PID 14398 died...
08:09:11.711521 run   osmo-bsc_10.42.42.8(pid=14398): Terminated: ok {rc=0}
08:09:14.903071 run          osmo-bts-trx(pid=15302): Terminating (SIGINT)
08:09:15.082178 ---      ParallelTerminationStrategy: PID 15302 died...
08:09:15.148245 run          osmo-bts-trx(pid=15302): Terminated: ok {rc=0}

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 367, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/ttcn3_bts_tests.py", line 53, in <module>
    run_ttcn3(tenv, testdir, bts, osmocon, nat_rsl_ip, ttcn3_test_groups)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 42, in run_ttcn3
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 39, in run_ttcn3
    proc.wait(timeout=5400) # 1.5h
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 367, in wait
    MainLoop.wait(self.terminated, timeout=timeout)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: ttcn3(pid=14417): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=14417)>, timeout=5400, timestep=1} [trial-2727↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=14417)]