|
Integrating a new subsystem into an existing
real-time system can be a formidable task. To facilitate any integration
effort, stringent stand-alone testing on the new subsystem needs to be
performed to verify compliance with the specifications defining the
interfaces/communication protocols and expected subsystem behavior
characteristics to interact with the rest of the system. However,
pre‑integration testing to determine the interoperability of a subsystem
often
fall short and many issues aren’t uncovered until integration tests
are conducted and are thus resolved much later in the development cycle,
invariably increasing cost and extending schedule. One of the issues
limiting the effectiveness of pre-integration testing is that such tasks are
normally manually intensive to perform and are at times not easily
repeatable. Another limiting factor is that such tests often take the
approach of developing a unique subsystem test set that can be fairly
expensive to build. While such test sets can be very effective, they are
often rigid and may not easily accommodate dynamically changing
requirements. Such time-consuming, manually-intensive test procedures can
be replaced with automated test coverage that remove the “man in the loop”
and provide a faster, more comprehensive test environment that can
induce/evaluate time critical and error related issues.
GBL has successfully researched and developed
automated test methodologies and implementations over the past 10 years for
the Navy on the F-14B Upgrade platform, validating interoperability
requirements on subsystems such as the Mission Computer (MC) and the
Controls & Displays Navigation Unit (CDNU). The benefits resulting from
having automated tests have greatly enhanced the overall software
development process for the F-14B Upgrade organization. The automated tests
developed enabled fuller coverage by exercising error paths in addition to
normal conditions and provided a readily repeatable regression test
environment that accelerated software releases when fixes were implemented.
Another benefit of establishing automated tests is
having readily repeatable regression testing that can be performed on
demand. The importance of this became apparent when GBL supported the Joint
Direct Attack Munition (JDAM) integration effort onto the F-14B Upgrade. In
one instance, a Mission Computer (MC) Operational Flight Software (OFS)
update was deemed necessary to fix incorrect A/C Environmental data being
supplied to the JDAM. The software update was incorporated just the day
before the scheduled first drop but the automated test drivers were able to
validate MC functionalities within a few hours, leaving enough time to
perform Safety of Flight tests and enable the JDAM drop to proceed as
planned.
Automated Test Experience
GBL has had extensive experience in developing
automated tests verifying the operation of complex real-time software
developed for the F-14D and F-14B Upgrade platforms. During development of
the F -14D Mission Computer Executive Program (FEP), GBL developed a testing
scheme that dynamically generated stress test cases and expected results
based on statistical modeling methods. These tests were executed
automatically around the clock to uncover stress related deficiencies and
thus helped to quickly stabilize the new operating system being developed.
The major advantage gained in performing such stringent testing was that
minimal FEP software updates were required due to stability issues after the
operating system was released.
At the onset of the F -14A/B Upgrade program, the
Naval Air Warfare Center (NAWC) tasked GBL to develop a Formal Quality Test
(FQT) for the Mission Computer Operational Flight Program (OFP). NAWC wanted
a comprehensive test that would exercise all the functional requirements of
the new Mission Computer as specified in the MIL-STD-2167A Software
Requirements Specification (SRS) and Interface Requirements Specification
(IRS) documents. GBL was responsible for defining the requirements for the
establishment of the F-14B Upgrade Subsystem Development Lab (SDL),
identifying all the necessary assets required to perform subsystem software
development and stand-alone automated tests. Prior to the development of
this environment, subsystem test were limited in scope or were very manually
intensive, requiring a substantial amount of time to complete end to end
test cycles. The development of an automated multi‑processor real‑time
simulation by GBL for the F-14A/B upgrade avionics system to test and
control the real F-14A/B Upgrade Mission Computer (real hardware in the
loop) through its multiple interfaces enabled FQTs to be completed within a
few hours, fully validating all operational requirements. This is in stark
contrast to the days/weeks other subsystems had to undergo FQTs that weren’t
even as comprehensive as the one developed for the Mission Computer. This
test and control simulation system was able to repeatedly test all the
different OFP requirements in just a few hours and provided a regression
baseline to test against for subsequent software releases.
Since the first test and control simulation
system software effort was much larger then the programming effort of the
OFP itself, GBL quickly developed a new approach to minimize system
testing. GBL developed a proprietary scripting language approach of
manipulating the existing test and control simulation software which
minimized the effort to create individual test cases. The addition of the
high level scripting language to the simulation system software enabled GBL
to rapidly and consistently test new OFP versions and to create new
scenarios for atypical circumstances. This scripting test and control
technique produced a more thorough test environment for checking many error
paths that weren’t succinctly specified by requirements. The new script
approach enabled the timely integration of the Embedded GPS/INS (EGI)
subsystem onto the F-14B Upgrade, even after a tremendous amount of the
schedule was impacted by encountered software development problems.
The addition of the high level scripting language
to the testing environment played a critical role in an accelerated effort
to integrate the Joint Direct Attack Munitions (JDAM) onto the F-14B Upgrade
platform. Only eighteen months elapsed from the onset of system design to
the first flight test JDAM drop. GBL developed automated tests that quickly
analyzed substantial amounts of MIL-STD-1553 message traffic and isolated
problem areas. Resulting output files enabled thorough checks to be
performed that ensured all mapped EGI elements were relayed to the JDAMs
along with the integral velocities/integral velocity corrections internally
calculated by the Mission Computer OFS, and that essential time critical
relationships were maintained. The automated tests also allowed quick
verification that mission data transfers and target assignments to the
weapons corresponded to what was expected.
Recently, GBL was tasked to re-host its high
level scripting language architecture to rapidly test and evaluate new
functionality of the Control and Display Navigation Unit (CDNU) for the
F -14A/B aircraft. The effort to re-host the test architecture developed for
the MC to perform a similar automated stand-alone subsystem tests for the
CDNU took only three months to complete and implement. Immediately, the
CDNU script driven automated test became immensely beneficial in verifying
all the CDNU’s requirements for each new CDNU software release, uncovering
CDNU communication and processing deficiencies. Problems were easily
isolated and resolved with the availability of repeatable tests duplicating
error conditions. New test cases were also readily generated to handle
dreaded requirements creep imposed during the integration timeline.
In addition to developing autonomous test
software, GBL also developed software to support post-run analysis on
collected data. Such software was used to validate the tightly coupled
timing relationships in the transfer alignment function required to
integrate the JDAM onto the F-14B Upgrade. These tools provided the means
to determine bus utilization, quickly decipher vast amounts of data,
validate the results of new algorithms, organize information via tables or
graphs, and pinpoint ambiguities for testers to focus additional attention
on.
Automated Test Architecture
The automated test architectures implemented by
GBL for the F-14B Upgrade used a “black-box” approach in developing
automated tests, focusing on the validation of a subsystem’s digital and
discrete interaction with the rest of the avionics system.
These architectures revolved around toolsets
capable of: (1) interacting over specified real-time interfaces, (2) rapidly
developing and stimulating actions/services of subsystem “proxies”, (3)
evaluating the responses of the subsystem/system under test, and (4)
executing automated test cases based on a scripting scheme that facilitates
step by step control over entire test sequences. The test software
developed automatically deciphered/interpreted all data transfers for
evaluation purposes and posted pertinent results/status for display to the
test operator. Current results and the actual step(s) that any particular
test case had failed were recorded into a database accessible by other
applications for subsequent analysis. Use of subsystem simulations make it
possible to generate script commands that inject error conditions into test
scenarios, thus allowing automated evaluation of error paths that are not
normally or easily exercised within existing integration environments.
Successful generation and verification of scripted test cases subsequently
provides a repeatable, automated regression test capability that is
independent of the expertise of test personnel.
Figure 1 –
Sample Automated Test Architecture
As no other subsystem other than the one under
test will be initially involved in pre-integration testing, the automated
test architecture uses subsystem simulations that can perform the
services/actions required to evaluate the behavior of the new subsystem.
While not intended to be form fit replacements of actual subsystems,
developed simulations must fully support all the services/actions required
to interact with the new subsystem and adhere to a mechanism under which the
services/actions are controllable by a Test Control toolset. GBL developed
generic templates which facilitated rapid development of new subsystem
simulations as they were needed.
Using a scripting approach, the Test Control
toolset acts as a high level script interpreter and is responsible for
initiating required actions by other components as defined for each script
command. As the central component overseeing the execution of scripted test
cases, the Test Control toolset has the capability of manipulating the
subsystem simulations as dictated by the test scenarios developed from the
subsystem specifications. In order to execute error cases, the Test Control
toolset has the capacity to disable or disrupt normal
activities/functionality of the subsystem simulations. The subsystem
simulations are responsible for reporting real-time status of the parameters
to be evaluated. Evaluation of the reactions of the subsystem under test
is then performed by the Test Control toolset to verify compliance with the
subsystem specifications. By utilizing a scripting scheme, test cases can
become self-documenting and easily altered to assess different logic paths
without necessitating the need to continuously update test software. This
allows step by step tracking of the progression of test cases via the
performed script commands to readily isolate detected problems as they are
found and also identify the conditions under which errors occur. |
