Enterprises routinely hand critical technology work to outside providers. Managed service providers patch systems, remediate vulnerabilities, monitor security events and perform other operational controls on behalf of their customers.
But outsourcing the work leaves CIOs with a fundamental question: How do they know the provider actually did what it was supposed to do?
In an Oct. 6 Reuters report, the news agency described the FBI's removal of a contractor after a data breach involving a platform managed by a third party. FBI cyber chief Brett Leatherman said the contractor had failed to apply a security patch. Reuters' sources identified the platform as Oracle PeopleSoft and the third-party organization as Accenture.
The incident illustrates a broader issue for enterprises: Assigning responsibility for operational work to an outside provider does not by itself establish that the work was completed.
Dashboards, monthly reports and closed tickets are summaries, not proof.
Adam Marrè
CISO, Arctic Wolf
Contracts establish responsibility and service-level agreements define expectations. Provider dashboards, tickets and reports can indicate that work was completed. But those records do not necessarily demonstrate that the underlying control was executed successfully or produced the intended result.
For CIOs, requiring evidence has practical limits. Independently checking every action could mean recreating the functions they hired a provider to perform.
"Dashboards, monthly reports and closed tickets are summaries, not proof," Adam Marrè, CISO at Arctic Wolf, told TechTarget.
For outsourced security operations, credible evidence can include timestamped records of actions taken, affected systems, the people or automation involved and the resulting change or recommendation, according to Marrè. Customers should be able to trace work from detection through resolution.
What counts as sufficient evidence depends on the work being performed.
Performing a control isn't the same as proving it worked
Vulnerability management provides a straightforward example.
Running a vulnerability scan shows that an activity occurred. It does not establish that critical vulnerabilities identified by the scan were prioritized and remediated, Marrè said.
For patching or vulnerability remediation, stronger evidence could include post-remediation vulnerability scan results, configuration validation and verification of the software version running on the affected asset. Comparing the state of an asset before and after remediation can help establish that the action succeeded rather than merely showing that it was initiated.
Auditors make a similar distinction between evidence that a control was performed and evidence that it operated effectively.
The sources of the information must always be validated before an auditor can rely on these types of end results as evidence.
Yvonne Sing
Principal Research Analyst for Audit Professional Practices, ISACA
A walkthrough can show how a control is performed at a particular point in time, said Yvonne Sing, CISA, CDPSE, principal research analyst for audit professional practices at ISACA. But one occurrence does not establish that the control operated effectively throughout the review period.
Effectiveness requires evidence that the control operated consistently during the period being reviewed and was performed by people with the appropriate authority, Sing said.
Provider dashboards and closed tickets present another evidence problem.
Those records represent the end result of a control process, Sing said. They do not necessarily show the inputs behind that result, including the underlying data and decisions made by the people operating the control.
"The sources of the information must always be validated before an auditor can rely on these types of end results as evidence," Sing told TechTarget.
Verification doesn't mean checking everything
Requiring evidence doesn't mean an enterprise has to independently verify every task its provider performs.
"Security teams shouldn't need to independently verify every action, but they should regularly test to make sure security controls are producing the expected results," Marrè said.
The amount of additional testing should instead reflect the risk to the business and the organization's tolerance for failure, according to Sing.
Banks, healthcare organizations and other heavily regulated businesses may have less tolerance for control failures because of potential regulatory, financial and reputational consequences. Those organizations might conduct independent testing in addition to reviewing third-party attestation reports.
Enterprises can also look across reporting periods rather than evaluate provider evidence in isolation. Comparing current and previous attestation reports can reveal recurring control exceptions or new exceptions in areas that previously had none, Sing said. Those patterns can help identify where a provider's control environment might be weakening and where additional testing is warranted.
Evidence requirements start with the contract
Waiting until an audit, control failure or security incident to request proof can make it harder to establish what a provider was expected to deliver.
Evidence expectations can instead be established at the start of the provider relationship.
Marrè said contracts should address access to supporting records, retention periods for logs and investigation artifacts, reporting of missed service levels or unresolved issues and the customer's ability to conduct reasonable audits.
Sing said useful evidence should contain enough information to establish its source and context. Depending on the process, that can include the name and title of the person providing the evidence, system name and version, system-generated date and time, report parameters, required approvals and the period the evidence covers.
Providers may resist broad audit rights because of concerns about disruption to daily operations, Sing said. One alternative is to negotiate audit rights covering higher-risk controls or business processes.
The evidence chain also needs to account for subcontractors.
When a primary provider relies on subservice organizations, contracts can require disclosure of the work those organizations perform, third-party attestation reports, access to evidence and information about how the primary provider assesses and addresses a subcontractor's failures, according to Sing.
CIOs don't need to independently check every action an outside provider takes. But when a critical control is outsourced, the enterprise still needs enough evidence to show that the work happened and produced the expected result.
Liz Hughes is an award-winning editor and writer covering AI and emerging technology and the former editor of AI Business and IoT World Today.
