description icon green arrowToday, government agencies, solution providers, and open standards organizations are developing standards and solutions that automate testing that verifies conformance to standards. Testing for code development can be done in-house but testing for conformance to the standards can be very challenging because the standards don’t come with the detailed conformance criteria or how-to adoption guidance. Demonstrating conformance to the implemented standards is important, and while manual tests and quality assurance are still necessary, automated tests provide consistent and reliable protection against unintentional regressions, and make it possible for developers to confidently release frequent updates.

 

Desired Outcome: Automated testing of systems against requirements and conformance to standards is in place and used to test various component systems as a part of the IS&S Environment.


checkmark logoPlay Checklist

 

1. Run usability tests frequently to see how well the IS&S Environment components works and identify improvements that should be made.
2. Conduct a pilot test of the use of the product in a scenario bearing resemblance to the original scenario (use case). This is sometimes called an alpha test of the product and must be realistic and result in actual implementation even if temporary. Evaluate results of the pilot.
3. Conduct a replication test where three to five other organizations not participating in the original development actually implement the product and test it in the real world. Evaluate the results of the replication process.
4. In parallel with the testing processes, prepare a documentation plan and package that includes release notes, user documentation, marketing collateral, and promotional materials such as video testimony.
5. If possible, work with software providers whose software products are certified to be in conformance with the standards adopted for the project.
6. Insist on the creation of a capability to automate software testing to allow for frequent retesting under load conditions to show conformance.
7. Require service providers to certify their products using the IJIS Institute Springboard Certification Program.

key questions icon purple question markPlay Key Questions

 

1. At each stage of usability testing, which gaps were identified in addressing user needs?
2. Has the software proposed been certified to conform to selected standards?
3. What tools are available for automating software testing?
4. What software product certifications does the software provider maintain for the proposed products?
5. Has the software provider employed automated testing in the past and what were the results?

resources icon yellow star Play Resources

 

The following resources can help you with this play:

 

Compliance Testing and Certification Programs such as: Springboard Certification Programhttp://www.ijis.org/?page=Springboard, and Open Geospatial Consortium (OGC) Compliance Testing and Certificationhttp://www.opengeospatial.org/compliance

 

 Information Sharing Environment (ISE) Information Interoperability Framework (I2F), http://ise.gov/sites/default/files/FINAL%20-%20ISE_I2F_v0%205.pdf

 

 Data Aggregation Reference Architecture (DARA), http://www.standardscoordination.org/sites/default/files/docs/DARA_v1.pdf

 

 Geospatial Interoperability Reference Architecture (GIRA), http://ise.gov/sites/default/files/GIRA.pdf


Return to the IS&S Environment Playbook