Skip to content

Regression

ttcn3_bts_tests:trx.ttcn3_bts_tests.py (from ttcn3_bts_tests_trx)

Failing for the past 1 build (Since #2751 )
Took 1 hr 30 min.

Stacktrace

ttcn3(pid=10468): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=10468)>, timeout=5400, timestep=1} [trial-2751↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=10468)]

Standard Output

----------------------------------------------
trial-2751 ttcn3_bts_tests:trx ttcn3_bts_tests.py
----------------------------------------------
00:21:31.747586 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.761454 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.775379 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.789715 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.803660 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.817681 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.831284 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:31.845455 tst              ttcn3_bts_tests:trx: Using 1 x bts (candidates: 1)
00:21:32.398979 tst               ttcn3_bts_tests.py: using LAC 9835
00:21:32.526689 tst               ttcn3_bts_tests.py: using RAC 145
00:21:32.654915 tst               ttcn3_bts_tests.py: using CellId 9834
00:21:32.778871 tst               ttcn3_bts_tests.py: using BVCI 2117
00:21:32.802913 tst              ttcn3_bts_tests:trx: Using 1 x osmocon_phone (candidates: 1)
00:21:32.988771 tst            ttcn3_bts_tests.py:35: Starting CNI
00:21:33.128987 run              osmo-hlr_10.42.42.2: Starting osmo-hlr
00:21:35.010855 run         create_hlr_db(pid=10437): Launched
00:21:36.201574 run         create_hlr_db(pid=10437): Terminated: ok {rc=0}
00:21:36.535797 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10439): Launched
00:21:36.861757 run   osmo-hlr_10.42.42.2(pid=10440): Launched
00:21:36.984182 run              osmo-stp_10.42.42.4: Starting osmo-stp
00:21:39.723431 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=10441): Launched
00:21:40.037998 run   osmo-stp_10.42.42.4(pid=10442): Launched
00:21:40.158221 run              osmo-msc_10.42.42.5: Starting osmo-msc
00:21:45.195195 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10443): Launched
00:21:45.509252 run   osmo-msc_10.42.42.5(pid=10444): Launched
00:21:45.635365 run              osmo-mgw_10.42.42.3: Starting osmo-mgw
00:21:47.489613 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=10445): Launched
00:21:47.816204 run   osmo-mgw_10.42.42.3(pid=10447): Launched
00:21:47.840433 tst              ttcn3_bts_tests:trx: Using 1 x ip_address (candidates: 8)
00:21:47.943160 run              osmo-bsc_10.42.42.8: Starting osmo-bsc
00:21:51.744066 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=10448): Launched
00:21:52.073548 run   osmo-bsc_10.42.42.8(pid=10449): Launched
00:21:52.190930 run                     osmo-bts-trx: Starting to connect to osmo-bsc_10.42.42.8
00:21:52.650005 run                     osmo-trx-uhd: OSMOTRX CONF: {'osmo_trx': {'dev_args': 'type=b200,serial=306BD11', 'bts_ip': '10.42.42.50', 'type': 'uhd', 'trx_ip': '10.42.42.116', 'multi_arfcn': 'True', 'remote_user': 'jenkins', 'clock_reference': 'external', 'egprs': 'disable', 'launch_trx': 'True', 'channels': [{}]}}
00:21:53.242408 run               uname-m(pid=10450): Launched
00:21:54.454329 run               uname-m(pid=10450): Terminated: ok {rc=0}
00:21:55.634641 run         rm-remote-dir(pid=10451): Launched
00:21:56.850758 run         rm-remote-dir(pid=10451): Terminated: ok {rc=0}
00:21:57.229384 run         mk-remote-dir(pid=10452): Launched
00:21:58.443562 run         mk-remote-dir(pid=10452): Terminated: ok {rc=0}
00:21:58.810238 run         mk-remote-dir(pid=10453): Launched
00:22:00.022429 run         mk-remote-dir(pid=10453): Terminated: ok {rc=0}
00:22:00.334626 run    scp-inst-to-remote(pid=10454): Launched
00:22:02.539380 run    scp-inst-to-remote(pid=10454): Terminated: ok {rc=0}
00:22:02.887037 run         mk-remote-dir(pid=10456): Launched
00:22:04.089989 run         mk-remote-dir(pid=10456): Terminated: ok {rc=0}
00:22:04.386946 run     scp-cfg-to-remote(pid=10457): Launched
00:22:05.588691 run     scp-cfg-to-remote(pid=10457): Terminated: ok {rc=0}
00:22:05.952547 run         mk-remote-dir(pid=10459): Launched
00:22:07.172555 run         mk-remote-dir(pid=10459): Terminated: ok {rc=0}
00:22:07.484902 run scp-wrapper-to-remote(pid=10460): Launched
00:22:08.691806 run scp-wrapper-to-remote(pid=10460): Terminated: ok {rc=0}
00:22:09.042033 run          osmo-trx-uhd(pid=10462): Launched
00:22:09.164042 run                     osmo-bts-trx: Waiting for osmo-trx-uhd to start up...
00:22:13.335418 run          osmo-bts-trx(pid=10463): Launched
00:22:13.538402 tst            ttcn3_bts_tests.py:49: Starting osmocon
00:22:13.682656 run                  osmocon_ttyUSB4: Resetting the phone
00:22:13.808291 run                  osmocon_ttyUSB4: Starting osmocon
00:22:15.532329 run       osmocon_ttyUSB4(pid=10464): Launched
00:22:15.660602 run                  osmocon_ttyUSB4: Waiting for osmocon to be up and running
00:22:16.859540 run          osmo-bts-trx(pid=10463): Terminated: ok {rc=0}
00:22:17.242496 run          osmo-bts-trx(pid=10466): Launched
00:22:17.485006 run          osmo-trx-uhd(pid=10462): Terminated: ok {rc=0}
00:22:17.848731 run          osmo-trx-uhd(pid=10467): Launched
00:22:18.047740 tst            ttcn3_bts_tests.py:53: Creating template
00:22:18.274903 tst            ttcn3_bts_tests.py:53: Starting TTCN3 test suite
00:22:18.577630 run                 ttcn3(pid=10468): Launched
00:22:18.809544 tst            ttcn3_bts_tests.py:53: TTCN3 test suite launched, waiting until it finishes
00:22:21.031735 run          osmo-bts-trx(pid=10466): Terminated: ok {rc=0}
00:22:21.446715 run          osmo-bts-trx(pid=10863): Launched
00:22:25.038763 run          osmo-bts-trx(pid=10863): Terminated: ok {rc=0}
00:22:25.409062 run          osmo-bts-trx(pid=10877): Launched
00:22:26.044650 run          osmo-trx-uhd(pid=10467): Terminated: ok {rc=0}
00:22:26.426645 run          osmo-trx-uhd(pid=10880): Launched
00:22:54.086240 run          osmo-bts-trx(pid=10877): Terminated: ok {rc=0}
00:22:54.450920 run          osmo-bts-trx(pid=10915): Launched
00:23:08.110892 run          osmo-bts-trx(pid=10915): Terminated: ok {rc=0}
00:23:08.476206 run          osmo-bts-trx(pid=10947): Launched
00:23:22.136034 run          osmo-bts-trx(pid=10947): Terminated: ok {rc=0}
00:23:22.499752 run          osmo-bts-trx(pid=10979): Launched
00:23:38.176748 run          osmo-bts-trx(pid=10979): Terminated: ok {rc=0}
00:23:38.546687 run          osmo-bts-trx(pid=11023): Launched
00:24:27.252748 run          osmo-bts-trx(pid=11023): Terminated: ok {rc=0}
00:24:27.618756 run          osmo-bts-trx(pid=11061): Launched
00:24:52.303865 run          osmo-bts-trx(pid=11061): Terminated: ok {rc=0}
00:24:52.678592 run          osmo-bts-trx(pid=11099): Launched
00:25:23.356456 run          osmo-bts-trx(pid=11099): Terminated: ok {rc=0}
00:25:23.728449 run          osmo-bts-trx(pid=11141): Launched
00:26:07.434224 run          osmo-bts-trx(pid=11141): Terminated: ok {rc=0}
00:26:07.806048 run          osmo-bts-trx(pid=11179): Launched
00:26:59.525474 run          osmo-bts-trx(pid=11179): Terminated: ok {rc=0}
00:26:59.896651 run          osmo-bts-trx(pid=11215): Launched
00:27:29.576389 run          osmo-bts-trx(pid=11215): Terminated: ok {rc=0}
00:27:29.952397 run          osmo-bts-trx(pid=11253): Launched
00:27:44.603731 run          osmo-bts-trx(pid=11253): Terminated: ok {rc=0}
00:27:44.991021 run          osmo-bts-trx(pid=11289): Launched
00:27:59.630734 run          osmo-bts-trx(pid=11289): Terminated: ok {rc=0}
00:28:00.007539 run          osmo-bts-trx(pid=11323): Launched
01:52:19.200427 run                 ttcn3(pid=10468): Terminating (SIGINT)
01:52:20.821584 run                 ttcn3(pid=10468): Terminated {rc=130}
01:52:20.858201 tst            ttcn3_bts_tests.py:53: ERR: Error: ttcn3(pid=10468): Wait timeout {condition=&lt;bound method Process.terminated of ttcn3(pid=10468)&gt;, timeout=5400, timestep=1} [trial-2751↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=10468)]  [trial-2751↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53]
01:52:20.869988 tst            ttcn3_bts_tests.py:53: Test FAILED (5449.2 sec)
01:52:20.923282 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10439): Terminating (SIGTERM)
01:52:20.977049 run   osmo-hlr_10.42.42.2(pid=10440): Terminating (SIGTERM)
01:52:21.032951 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=10441): Terminating (SIGTERM)
01:52:21.090270 run   osmo-stp_10.42.42.4(pid=10442): Terminating (SIGTERM)
01:52:21.150459 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10443): Terminating (SIGTERM)
01:52:21.208463 run   osmo-msc_10.42.42.5(pid=10444): Terminating (SIGTERM)
01:52:21.262194 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=10445): Terminating (SIGTERM)
01:52:21.316570 run   osmo-mgw_10.42.42.3(pid=10447): Terminating (SIGTERM)
01:52:21.371609 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=10448): Terminating (SIGTERM)
01:52:21.425427 run   osmo-bsc_10.42.42.8(pid=10449): Terminating (SIGTERM)
01:52:21.478653 run          osmo-trx-uhd(pid=10880): Terminating (SIGTERM)
01:52:21.532623 run          osmo-bts-trx(pid=11323): Terminating (SIGTERM)
01:52:21.585144 run       osmocon_ttyUSB4(pid=10464): Terminating (SIGTERM)
01:52:21.597290 ---      ParallelTerminationStrategy: PID 10439 died...
01:52:21.666610 run pcap-recorder_any(filters='host 10.42.42.2')(pid=10439): Terminated: ok {rc=0}
01:52:21.679348 ---      ParallelTerminationStrategy: PID 10440 died...
01:52:21.749095 run   osmo-hlr_10.42.42.2(pid=10440): Terminated: ok {rc=0}
01:52:21.761214 ---      ParallelTerminationStrategy: PID 10441 died...
01:52:21.828660 run pcap-recorder_any(filters='host 10.42.42.4 and port not 22')(pid=10441): Terminated: ok {rc=0}
01:52:21.840813 ---      ParallelTerminationStrategy: PID 10442 died...
01:52:21.908193 run   osmo-stp_10.42.42.4(pid=10442): Terminated: ok {rc=0}
01:52:21.920231 ---      ParallelTerminationStrategy: PID 10443 died...
01:52:21.993311 run pcap-recorder_any(filters='host 10.42.42.5 and port not 22')(pid=10443): Terminated: ok {rc=0}
01:52:22.005545 ---      ParallelTerminationStrategy: PID 10445 died...
01:52:22.073961 run pcap-recorder_any(filters='host 10.42.42.3 and port not 22')(pid=10445): Terminated: ok {rc=0}
01:52:22.086166 ---      ParallelTerminationStrategy: PID 10447 died...
01:52:22.154537 run   osmo-mgw_10.42.42.3(pid=10447): Terminated {rc=15}
01:52:22.166638 ---      ParallelTerminationStrategy: PID 10448 died...
01:52:22.234544 run pcap-recorder_any(filters='host 10.42.42.8 or host 10.42.42.9 and port not 22')(pid=10448): Terminated: ok {rc=0}
01:52:22.246751 ---      ParallelTerminationStrategy: PID 10464 died...
01:52:22.314489 run       osmocon_ttyUSB4(pid=10464): Terminated {rc=15}
01:52:22.326627 ---      ParallelTerminationStrategy: PID 10880 died...
01:52:22.395332 run          osmo-trx-uhd(pid=10880): Terminated {rc=36608}
01:52:24.424301 ---      ParallelTerminationStrategy: PID 10444 died...
01:52:24.496688 run   osmo-msc_10.42.42.5(pid=10444): Terminated: ok {rc=0}
01:52:25.175954 ---      ParallelTerminationStrategy: PID 10449 died...
01:52:25.245375 run   osmo-bsc_10.42.42.8(pid=10449): Terminated: ok {rc=0}
01:52:28.438028 run          osmo-bts-trx(pid=11323): Terminating (SIGINT)
01:52:28.617773 ---      ParallelTerminationStrategy: PID 11323 died...
01:52:28.686946 run          osmo-bts-trx(pid=11323): Terminated: ok {rc=0}

Standard Error

Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/test.py", line 76, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/util.py", line 414, in run_python_file
    spec.loader.exec_module( importlib.util.module_from_spec(spec) )
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/ttcn3_bts_tests.py", line 53, in <module>
    run_ttcn3(tenv, testdir, bts, osmocon, nat_rsl_ip, ttcn3_test_groups)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 42, in run_ttcn3
    raise e
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/sysmocom/ttcn3/suites/ttcn3_bts_tests/lib/testlib.py", line 39, in run_ttcn3
    proc.wait(timeout=5400) # 1.5h
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/process.py", line 367, in wait
    MainLoop.wait(self.terminated, timeout=timeout)
  File "/home/jenkins/workspace/osmo-gsm-tester_ttcn3/osmo-gsm-tester/src/osmo_gsm_tester/core/event_loop.py", line 109, in wait
    raise log.Error('Wait timeout', condition=condition, timeout=timeout, timestep=timestep)
osmo_gsm_tester.core.log.Error: ttcn3(pid=10468): Wait timeout {condition=<bound method Process.terminated of ttcn3(pid=10468)>, timeout=5400, timestep=1} [trial-2751↪ttcn3_bts_tests:trx↪ttcn3_bts_tests.py:53↪ttcn3(pid=10468)]