Skip to content

Regression

ttcn3_bts_tests:trx.ttcn3_bts_tests.py (from ttcn3_bts_tests_trx)

Failing for the past 1 build (Since #2754 )
Took 1 hr 30 min.

Stacktrace

ttcn3(pid=12129): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=12129)>, timeout=5400, timestep=1} [trial-2754↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=12129)]

Standard Output

----------------------------------------------
trial-2754 ttcn3_bts_tests:trx ttcn3_bts_tests.py
----------------------------------------------
00:21:12.514523 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.528654 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.542722 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.557440 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.571748 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.586357 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.600452 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:12.614728 tst              ttcn3_bts_tests:trx: Using 1 x bts (candidates: 1)
00:21:13.178554 tst               ttcn3_bts_tests.py: using LAC 9849
00:21:13.310219 tst               ttcn3_bts_tests.py: using RAC 159
00:21:13.438678 tst               ttcn3_bts_tests.py: using CellId 9848
00:21:13.567747 tst               ttcn3_bts_tests.py: using BVCI 2131
00:21:13.593477 tst              ttcn3_bts_tests:trx: Using 1 x osmocon_phone (candidates: 1)
00:21:13.782487 tst            ttcn3_bts_tests.py:35: Starting CNI
00:21:13.921001 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
00:21:15.713521 run         create_hlr_db(pid=12099): Launched
00:21:16.902889 run         create_hlr_db(pid=12099): Terminated: ok {rc=0}
00:21:17.233386 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12101): Launched
00:21:17.545926 run   osmo-hlr_10.42.42.2(pid=12102): Launched
00:21:17.658608 run              osmo-stp_10.42.42.4: Starting osmo-stp
00:21:20.384930 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=12103): Launched
00:21:20.698559 run   osmo-stp_10.42.42.4(pid=12104): Launched
00:21:20.811053 run              osmo-msc_10.42.42.5: Starting osmo-msc
00:21:25.782145 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12105): Launched
00:21:26.095079 run   osmo-msc_10.42.42.5(pid=12106): Launched
00:21:26.209097 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
00:21:28.030350 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=12107): Launched
00:21:28.344375 run   osmo-mgw_10.42.42.3(pid=12108): Launched
00:21:28.368694 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:28.471127 run              osmo-bsc_10.42.42.8: Starting osmo-bsc
00:21:32.336479 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=12109): Launched
00:21:32.670278 run   osmo-bsc_10.42.42.8(pid=12110): Launched
00:21:32.794658 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.8
00:21:33.258270 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'trx_ip': '10.42.42.116', 'bts_ip': '10.42.42.50', 'channels': [{}], 'egprs': 'disable', 'type': 'uhd', 'dev_args': 'type=b200,serial=306BD11', 'clock_reference': 'external', 'remote_user': 'jenkins', 'multi_arfcn': 'True', 'launch_trx': 'True'}}
00:21:33.847761 run               uname-m(pid=12111): Launched
00:21:35.066873 run               uname-m(pid=12111): Terminated: ok {rc=0}
00:21:36.410367 run         rm-remote-dir(pid=12112): Launched
00:21:37.616595 run         rm-remote-dir(pid=12112): Terminated: ok {rc=0}
00:21:37.979990 run         mk-remote-dir(pid=12113): Launched
00:21:39.184110 run         mk-remote-dir(pid=12113): Terminated: ok {rc=0}
00:21:39.531299 run         mk-remote-dir(pid=12114): Launched
00:21:40.733269 run         mk-remote-dir(pid=12114): Terminated: ok {rc=0}
00:21:41.038350 run    scp-inst-to-remote(pid=12115): Launched
00:21:43.253372 run    scp-inst-to-remote(pid=12115): Terminated: ok {rc=0}
00:21:43.615106 run         mk-remote-dir(pid=12117): Launched
00:21:44.827231 run         mk-remote-dir(pid=12117): Terminated: ok {rc=0}
00:21:45.145330 run     scp-cfg-to-remote(pid=12118): Launched
00:21:46.359562 run     scp-cfg-to-remote(pid=12118): Terminated: ok {rc=0}
00:21:46.734669 run         mk-remote-dir(pid=12120): Launched
00:21:47.952587 run         mk-remote-dir(pid=12120): Terminated: ok {rc=0}
00:21:48.266391 run scp-wrapper-to-remote(pid=12121): Launched
00:21:49.481421 run scp-wrapper-to-remote(pid=12121): Terminated: ok {rc=0}
00:21:49.835654 run          osmo-trx-uhd(pid=12123): Launched
00:21:49.958272 run                     osmo-bts-trx: Waiting for osmo-trx-uhd to start up...
00:21:54.105500 run          osmo-bts-trx(pid=12124): Launched
00:21:54.310132 tst            ttcn3_bts_tests.py:49: Starting osmocon
00:21:54.451897 run                  osmocon_ttyUSB4: Resetting the phone
00:21:54.576296 run                  osmocon_ttyUSB4: Starting osmocon
00:21:56.232308 run       osmocon_ttyUSB4(pid=12125): Launched
00:21:56.363513 run                  osmocon_ttyUSB4: Waiting for osmocon to be up and running
00:21:57.564012 run          osmo-bts-trx(pid=12124): Terminated: ok {rc=0}
00:21:57.932779 run          osmo-bts-trx(pid=12127): Launched
00:21:58.174834 run          osmo-trx-uhd(pid=12123): Terminated: ok {rc=0}
00:21:58.546715 run          osmo-trx-uhd(pid=12128): Launched
00:21:58.761961 tst            ttcn3_bts_tests.py:53: Creating template
00:21:59.011135 tst            ttcn3_bts_tests.py:53: Starting TTCN3 test suite
00:21:59.320554 run                 ttcn3(pid=12129): Launched
00:21:59.534466 tst            ttcn3_bts_tests.py:53: TTCN3 test suite launched, waiting until it finishes
00:22:01.829802 run          osmo-bts-trx(pid=12127): Terminated: ok {rc=0}
00:22:02.409272 run          osmo-bts-trx(pid=12530): Launched
00:22:05.757987 run          osmo-bts-trx(pid=12530): Terminated: ok {rc=0}
00:22:06.123412 run          osmo-bts-trx(pid=12544): Launched
00:22:06.367956 run          osmo-trx-uhd(pid=12128): Terminated: ok {rc=0}
00:22:06.721657 run          osmo-trx-uhd(pid=12545): Launched
00:22:33.965710 run          osmo-bts-trx(pid=12544): Terminated: ok {rc=0}
00:22:34.342247 run          osmo-bts-trx(pid=12579): Launched
00:22:47.992043 run          osmo-bts-trx(pid=12579): Terminated: ok {rc=0}
00:22:48.367437 run          osmo-bts-trx(pid=12609): Launched
00:23:02.008736 run          osmo-bts-trx(pid=12609): Terminated: ok {rc=0}
00:23:02.374444 run          osmo-bts-trx(pid=12641): Launched
00:23:18.035588 run          osmo-bts-trx(pid=12641): Terminated: ok {rc=0}
00:23:18.400090 run          osmo-bts-trx(pid=12685): Launched
00:24:07.134035 run          osmo-bts-trx(pid=12685): Terminated: ok {rc=0}
00:24:07.506737 run          osmo-bts-trx(pid=12723): Launched
00:24:32.173794 run          osmo-bts-trx(pid=12723): Terminated: ok {rc=0}
00:24:32.546183 run          osmo-bts-trx(pid=12761): Launched
00:25:03.218775 run          osmo-bts-trx(pid=12761): Terminated: ok {rc=0}
00:25:03.584152 run          osmo-bts-trx(pid=12802): Launched
00:25:25.255820 run          osmo-bts-trx(pid=12802): Terminated: ok {rc=0}
00:25:25.610931 run          osmo-bts-trx(pid=12835): Launched
00:26:18.355640 run          osmo-bts-trx(pid=12835): Terminated: ok {rc=0}
00:26:18.728879 run          osmo-bts-trx(pid=12871): Launched
00:26:48.395021 run          osmo-bts-trx(pid=12871): Terminated: ok {rc=0}
00:26:48.749982 run          osmo-bts-trx(pid=12909): Launched
00:27:03.429260 run          osmo-bts-trx(pid=12909): Terminated: ok {rc=0}
00:27:03.809438 run          osmo-bts-trx(pid=12941): Launched
00:27:18.445598 run          osmo-bts-trx(pid=12941): Terminated: ok {rc=0}
00:27:18.810422 run          osmo-bts-trx(pid=12974): Launched
00:28:30.572996 run          osmo-bts-trx(pid=12974): Terminated: ok {rc=0}
00:28:30.947086 run          osmo-bts-trx(pid=13009): Launched
01:52:00.089849 run                 ttcn3(pid=12129): Terminating (SIGINT)
01:52:01.719193 run                 ttcn3(pid=12129): Terminated {rc=130}
01:52:01.754850 tst            ttcn3_bts_tests.py:53: ERR: Error: ttcn3(pid=12129): Wait timeout {condition=&lt;bound method Process.terminated of ttcn3(pid=12129)&gt;, timeout=5400, timestep=1} [trial-2754↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=12129)]  [trial-2754↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53]
01:52:01.766653 tst            ttcn3_bts_tests.py:53: Test FAILED (5449.3 sec)
01:52:01.814521 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12101): Terminating (SIGTERM)
01:52:01.869407 run   osmo-hlr_10.42.42.2(pid=12102): Terminating (SIGTERM)
01:52:01.925880 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=12103): Terminating (SIGTERM)
01:52:01.984202 run   osmo-stp_10.42.42.4(pid=12104): Terminating (SIGTERM)
01:52:02.042110 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12105): Terminating (SIGTERM)
01:52:02.100656 run   osmo-msc_10.42.42.5(pid=12106): Terminating (SIGTERM)
01:52:02.158205 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=12107): Terminating (SIGTERM)
01:52:02.212178 run   osmo-mgw_10.42.42.3(pid=12108): Terminating (SIGTERM)
01:52:02.266675 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=12109): Terminating (SIGTERM)
01:52:02.320548 run   osmo-bsc_10.42.42.8(pid=12110): Terminating (SIGTERM)
01:52:02.374434 run          osmo-trx-uhd(pid=12545): Terminating (SIGTERM)
01:52:02.426917 run          osmo-bts-trx(pid=13009): Terminating (SIGTERM)
01:52:02.477935 run       osmocon_ttyUSB4(pid=12125): Terminating (SIGTERM)
01:52:02.489564 ---      ParallelTerminationStrategy: PID 12101 died...
01:52:02.558256 run pcap-recorder_any(filters='host 10.42.42.2')(pid=12101): Terminated: ok {rc=0}
01:52:02.570951 ---      ParallelTerminationStrategy: PID 12102 died...
01:52:02.639998 run   osmo-hlr_10.42.42.2(pid=12102): Terminated: ok {rc=0}
01:52:02.652005 ---      ParallelTerminationStrategy: PID 12103 died...
01:52:02.719633 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=12103): Terminated: ok {rc=0}
01:52:02.731714 ---      ParallelTerminationStrategy: PID 12104 died...
01:52:02.798732 run   osmo-stp_10.42.42.4(pid=12104): Terminated: ok {rc=0}
01:52:02.810811 ---      ParallelTerminationStrategy: PID 12105 died...
01:52:02.878232 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=12105): Terminated: ok {rc=0}
01:52:02.890321 ---      ParallelTerminationStrategy: PID 12107 died...
01:52:02.962880 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=12107): Terminated: ok {rc=0}
01:52:02.974965 ---      ParallelTerminationStrategy: PID 12108 died...
01:52:03.042310 run   osmo-mgw_10.42.42.3(pid=12108): Terminated {rc=15}
01:52:03.054446 ---      ParallelTerminationStrategy: PID 12109 died...
01:52:03.122109 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=12109): Terminated: ok {rc=0}
01:52:03.134322 ---      ParallelTerminationStrategy: PID 12125 died...
01:52:03.202100 run       osmocon_ttyUSB4(pid=12125): Terminated {rc=15}
01:52:03.214305 ---      ParallelTerminationStrategy: PID 12545 died...
01:52:03.282467 run          osmo-trx-uhd(pid=12545): Terminated {rc=36608}
01:52:05.311485 ---      ParallelTerminationStrategy: PID 12106 died...
01:52:05.380859 run   osmo-msc_10.42.42.5(pid=12106): Terminated: ok {rc=0}
01:52:06.059940 ---      ParallelTerminationStrategy: PID 12110 died...
01:52:06.128275 run   osmo-bsc_10.42.42.8(pid=12110): Terminated: ok {rc=0}
01:52:09.321024 run          osmo-bts-trx(pid=13009): Terminating (SIGINT)
01:52:09.500593 ---      ParallelTerminationStrategy: PID 13009 died...
01:52:09.568969 run          osmo-bts-trx(pid=13009): Terminated: ok {rc=0}

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/ttcn3_bts_tests.py", line 53, in <module>
    run_ttcn3(tenv, testdir, bts, osmocon, nat_rsl_ip, ttcn3_test_groups)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 42, in run_ttcn3
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 39, in run_ttcn3
    proc.wait(timeout=5400) # 1.5h
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 367, in wait
    MainLoop.wait(self.terminated, timeout=timeout)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: ttcn3(pid=12129): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=12129)>, timeout=5400, timestep=1} [trial-2754↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=12129)]