Legal basis for collecting and storing logs without consent

You can’t collect and store any personal data without having obtained, and being able to document that you obtained, consent from the persons you’re collecting data from. You can, however, still collect and store personal data in your server logs for the limited and legitimate purpose of detecting and preventing fraud and unauthorized system access, and ensuring the security of your systems.

Here are the relevant excerpts from the GDPR that allows data collection for this type of purposes:

“Processing shall be lawful only if and to the extent that at least one of the following applies: […] (f) processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data, in particular where the data subject is a child.”

Article 6, Paragraph 1, Point F

“The processing of personal data to the extent strictly necessary and proportionate for the purposes of ensuring network and information security, i.e. the ability of a network or an information system to resist, at a given level of confidence, accidental events or unlawful or malicious actions that compromise the availability, authenticity, integrity and confidentiality of stored or transmitted personal data, and the security of the related services offered by, or accessible via, those networks and systems, […] by providers of electronic communications networks and services and by providers of security technologies and services, constitutes a legitimate interest of the data controller concerned. This could, for example, include preventing unauthorised access to electronic communications networks and malicious code distribution and stopping ‘denial of service’ attacks and damage to computer and electronic communication systems.”

Recital 49 (excerpt)Notably, this doesn’t exempt such collection from the strict requirements of the GDPR. Gandalf the Grey offers some great advice for how you should treat personal data to achieve GDPR compliance in your organization:

Encryption, access restriction, and timely erasure

The specific requirements under the GDPR that apply to your organization depends on the scope and type of data you collect set against the needs to store the data. The regulation with all its recitals is 54 800 words long, but I’ll try to summarize some practical implementation requirements from the regulation:

There are no specific technical details offered regarding how these requirements should be implemented besides the suggestion to use “encryption” in Article 6, Paragraph 4, Point E and Recital 83. The take-away is still clear: data should be secured, access should be limited even within your organization, and data should be deleted (including from backups) when there is no longer a need to retain it.

Utility of the day: logrotate (+ gnupg)

logrotate is a very useful tool that can be used to encrypt logs in storage even on edge servers, and can help automate deletion of old log files. It can also be used to encrypt log files in storage which when combined with organizational measures can limit access to decrypting the log files.. Unless you encrypt your logs, an unauthorized third-party who gained access to your servers could extract a lot of data about your users from your logs. Depending on how much private information is stored in your log files and the potential sensitive nature of your business, you shouldn’t store log files for more than a few hours or days unless you take measures to protect them.

Managing PGP keys in GnuPG is beyond the scope of this article. However, in short you would create a key-pair on a secure machine, and then import the public key into the GnuPG key chain on your servers while storing the private key on a secure medium with limited access for authorized employees only (e.g. printed on paper or kept on a removable storage media). The server can then use the public key to encrypt its log files without with public key cryptography; resulting in the server being able to encrypt the data without being able to decrypt it without the private. The log files could even be transferred to a centralized log-storage server for cold storage.

I believe such a setup could be used to achieve GDPR compliance while still maintaining auditable logs in the event of breach of server security or other incident that would require a log trail.

The following logrotate configuration example demonstrates secure encrypted storage (using GnuPG) erasure after time intervals (rotate in days) appropriate to how important it is to store the various log files following a security incident.

In the above example, access logs are deleted after 100 days, error logs after 200 days, and ModSecurity logs (which would only contain suspicious activity), are retained for 400 days. After this time, the logs would be securely erased using the shred utility.

The logs are still kept unencrypted for up to 24-hours when they were first recorded. This is a small time window when the data is not stored encrypted, but it is required to allow human technicians and automated log analyzing tools (like SSHGuard or Fail2Ban) to process the data and act upon it to help detect and prevent unauthorized or unlawful system access.

You can reduce the time window when data is kept unencrypted by rotating logs hourly instead of daily, or by piping logs directly from your web server into an encrypted storage. This may have a serious performance impact on your server and complicate the configuration of automated security monitoring tools.

Summary

Any identifier, including network or equipment identifiers like an IP address, are considered personal data. Don’t store server logs if you don’t have to. Encrypt logs in storage and limit access to decryption credentials. Delete logs as early as possible, including from any backups. Document what steps you’ve taken to secure data and limit impact in the case of a server breach.

This article has focused on server logs as they’re something every organization with a website or an online service will have to deal with. However, the same principles and even stricter requirements apply to other types of data that your organization keeps on people. The deadline for GDPR compliance is May 25th 2018, and that is barely enough time to read through the 54 800 word regulation. With fines up in the 20 Million Euro range; you better get started auditing personal information collection and storage in your organization right away. This is the perfect time to rethink old decisions regarding what data your organization really needs to keep and for how long.