What happened
NYC DCAS's CAMS was described in the case evidence as operational while formal completion/acceptance and important verification remained unresolved.
The New York City Department of Citywide Administrative Services contracted for a web-based Capital Asset Management System (CAMS) covering 53 public buildings. The implementation included facility and infrastructure condition information intended to support capital planning.
CAMS was in operational use while formal acceptance remained unresolved, with review of the database information and outstanding software issues still open at the time of the audit.
Why it matters
Operational status and acceptance status can be different states.
Has the system entered operation? That is not necessarily the same question as: has the buyer established that what was promised was delivered to an acceptable standard?
The evidence problem
The New York City Comptroller attempted to test CAMS information using a sample of 10 buildings and sought supporting documentation. The information supplied was not sufficient for the auditors to independently substantiate the accuracy of the underlying CAMS data.
Our research interpretationSoftware acceptance and information acceptance are not necessarily the same thing.
What a buyer should do
Define acceptance before implementation reaches the finish line.
Define acceptance criteria, verification methods, evidence, tolerance and acceptance authority before acceptance is required; track unresolved verification.
Define acceptance evidence before the point at which acceptance is required.
Material requirements should have corresponding evidence of satisfaction. Depending on what the project actually involves, that evidence can include functional testing, data reconciliation, integration verification, realistic operating tests, security evidence, recovery testing or formal owner approval. Not every project requires every form of testing; the point is that each material requirement has a defined way of being shown to be met.