The European Commission is to evaluate a proposal to reduce child abuse through mechanisms to control material on users’ devices. This initiative has raised a number of controversies and comments that deserve attention.
Before examining this issue, it is considered important to make a few clarifications: the purpose of this article is not to declare what is right or wrong, this issue reaches an impressive level of legal complexity. It is mostly a legal rather than a technical issue. Finally, it is recommended to approach this issue with neutrality of mind: opinions should come after a careful, critical and impartial examination of the text of the proposal.
Context of reference
The ‘Chat Control’ / ‘Child Sexual Abuse Regulation’ (CSAR) is a proposal of the European Commission under reference COM(2022)209. The proposal aims to establish binding rules to prevent, combat and remove child sexual abuse material online (CSAM = Child Sexual Abuse Material). The legal basis of the proposal is Article 114 of the Treaty on the Functioning of the European Union (TFEU), which is used to regulate aspects of the internal market. By introducing an obligation for providers to detect, report, block and remove child sexual abuse material from their services, the proposal enables better detection, investigation and prosecution of offences under the directive on combating the sexual abuse of children. The legislation complements the new European Strategy for a Better Internet for Children, which aims to create safe digital experiences for children and promote digital empowerment and responsibility.
Essentially, the proposal is that national authorities can order communication service providers (e.g. messaging apps) to scan content circulating on their platform for CSAM material. This includes images, videos, URLs, text. Such scanning involves comparing suspicious material with that of authorities including that fed and managed by the NCMEC (National Center for Missing & Exploited Children) of the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. The so-called ‘affected information society services‘ (Art. 2) are impacted by the rule, namely:
- hosting service;
- interpersonal communication service;
- software application shop;
- internet access service;
Technical operation
At the basis of the mechanism is a comparison of the hash code (hash matching) between the original one held by the CNMEC and the one identified in the services used by users. The hash code comparison (hash matching) mechanism is not new; consider that Google has developed effective hash matching techniques, also in collaboration with CNMEC itself.
The system, therefore, does not foresee the circulation of the incriminated images between authorities and service providers, but compares the image footprint originally uploaded on the illicit site with the copies downloaded on their devices/cloud services by malicious users or spread through messaging systems. The problem that is contested with this solution is the pervasiveness that the system could have: it could be developed at the level of a single application or of the entire operating system.
If one reasoned only on the application level, smartphone manufacturers would have to remove all those apps that refuse to implement such a scanning and comparison engine. If one reasoned at the level of the entire operating system, the pervasiveness would be maximum: the entire smartphone (and not just the individual application) could be subjected to scanning. Hence the main criticism of the proposal: this mechanism, which today could be triggered for child abuse reasons, tomorrow could be used for mass surveillance programmes. The concern is therefore closely related to the privacy and freedom of users, along with the transparency of using a mechanism that, in itself, does not expose the material contained in users’ smartphones but only their fingerprints. It is a very delicate issue of freedom and legal principles, of rights compression and of current and future proportionality. In short, it is a major issue. From a technical point of view, it is plausible that the scanning capability would be introduced at the operating system level for obvious reasons: one has to think that a smartphone, when it is sold, already has applications for storing images, videos, texts and audio files. As well as tools for generating them: text composition, video recording, audio and photo capturing applications. Consequently, conceptually, the level of scanning of the entire operating system would make more logical sense, but would result in a total pervasiveness of the system.
Comparison mechanisms
From a technical point of view, many questioned the reliability of the proposed technique: basically, it would be enough to change the format, to crop the photo slightly, to fool a basic hash matching system. Indeed, the photos would not be the same and a comparison would produce different results. However, the system adopted would not be basic, there are partial analysis techniques, also based on artificial intelligence models, capable of making a sort of ‘conceptual comparison’ of the image.
A famous example is PhotoDNA developed by Microsoft in 2019. This application does not perform a ‘trivial comparison’ between hashes but analyses the image by comparing several parts of the image in search of a possible match.
This means that comparison would be possible even in the presence of non-destructive cropping and editing. In fact, the pixel composition of an image is itself a map of potential comparison even in the presence of small ‘anomalies’. By isolating a portion of the image (e.g. the face) and making a comparison between the original and the copy, it would then be possible to determine one or more points of compatibility. NCMEC already uses PhotoDNA as Microsoft has donated its technology to the organisation.
Microsoft donated PhotoDNA to the National Centre for Missing & Exploited Children (NCMEC).
There are also more complex mechanisms that, through the use of artificial intelligence models, perform more complex analogy analyses: poses, lighting, partial occlusion. In fact, some basic assumptions must be considered: however much an attacker may crop a photo, correct its colours, alter its pixels, the contents of main interest will always remain visible in the photo and on those the algorithm will be able to work.
The point of view of cloud service providers
The proposal motivates an important aspect concerning the service providers concerned.
This proposal aims to remove existing barriers to the provision of the services concerned within the Digital Single Market and to prevent the emergence of further barriers, while also enabling an effective fight against online child sexual abuse in full respect of the fundamental rights under Union law of all stakeholders. To achieve this objective, the proposal introduces strictly targeted and uniform risk assessment and mitigation obligations, complemented where necessary by orders to detect, report and remove child sexual abuse content. These obligations apply to providers offering services in the digital single market regardless of where they have their main establishment.
Hosting infringing material on one’s cloud is an offence in its own right and from the point of view of cloud service providers represents a difficult problem to manage. It would be absurd to expect manual and individual control of material stored by millions of users worldwide. It would also be absurd to tolerate a total lack of control. It is therefore clear that systems like PhotoDNA may be of interest to those offering cloud services. They automate the process, make it much faster and with a remarkable reliability index. In the case of Microsoft, integration with Azure makes it possible for companies that build and use cloud systems based on that infrastructure to maintain greater security.
In 2015, Microsoft made PhotoDNA available as a service on Azure. The PhotoDNA Cloud Service enables smaller companies and other organisations that want to give users the freedom to upload content while ensuring the integrity of their platforms.
However, it is good to make one thing clear right away: technically these systems are not infallible. However sophisticated they are, they may fail, especially in the presence of aggressive cropping, significant overlapping, geometric modifications, heavy filtering, or targeted perturbations (adversarial noise) that may reduce similarity below the detection threshold. They can also generate false positives on similar but harmless images. This, in particular, would be a serious problem because it would subject a non-infringing user to an inspection or control action for absolutely no reason.
The proposal includes a very interesting paragraph:
At the same time, the measures contained in the proposal primarily affect the exercise of the fundamental rights of the users of the services in question. These rights include in particular the fundamental rights to respect for private life (including the confidentiality of communications, as part of the broader right to respect for private and family life), to protection of personal data and to freedom of expression and information. Although of great importance, none of these rights is absolute, but must be considered in the light of its social function.
The European Commission therefore realises the privacy-violating nature to which the proposal exposes itself, but counters this risk with a rationale based on the general interest.
The proposal thus aims to achieve the general interest objective mentioned above and to protect these fundamental rights of children, while ensuring proportionality and striking a fair balance between the fundamental rights of all parties involved. To this end, it contains a number of limitations and safeguards, differentiated according to the nature and level of the limitation imposed on the exercise of the fundamental rights in question.
On the one hand, there is technology that aims to offer tools of relative impact for the scanning and detection of potentially infringing material. On the other, there is the law, which is faced with a challenge of no small countenance. Finally, there is the problem of false positives, which would undoubtedly be one of the most difficult aspects to manage. Imagining that the analysis tool fails (because there are no infallible tools) what would happen to the person wrongly flagged by the system? Would it make sense to introduce scanning systems that might have an inadequate level of accuracy? Furthermore, the awareness of the malicious person must be taken into account. Those who normally carry out this type of illicit action avoid having the material with them in ‘common’ devices and media. He avoids, in essence, exposing such files to control and surveillance systems, by adopting crafty means of possession and transfer (through channels other than Whatsapp, Telegram or similar). The less aware are those, on the other hand, who betray the security mechanisms by adopting these tools, and for whom systems such as those proposed are far too excessive precisely because they are careless and prone to errors. Beyond the more technical aspects, the proposal raises a relevant issue.
The impact assessment shows that the provision of voluntary actions against online child sexual abuse alone is insufficient, as only a limited number of providers adhere to it, and the challenges encountered in public-private cooperation are no less significant than the difficulties of Member States in preventing the phenomenon and ensuring an adequate level of assistance to victims.
Hence, prevention as well as law enforcement would reportedly be insufficient without collective and organised action.
How the detection takes place
According to the proposal in Article 7(3), if there is evidence of a significant risk of the service being used for online child sexual abuse, the coordinating authority:
- draws up a draft request for the issuance of a takeover order, in which it specifies the main elements of the content of the order it intends to request and the reasons for the request;
- submits the draft application to the lender and the EU Centre;
- gives the provider the opportunity to comment on the draft application within a reasonable time limit set by it;
- invites the EU Centre to issue an opinion on the draft request within four weeks from the date of its receipt.
The action would therefore be based on a proven risk analysed in a process hypothesised not to exclude executive ‘automatism’. It would, in essence, be a form of guarantee against a principle of abuse of the investigation mechanism. However, it should be pointed out that the subsequent subparagraphs of Article 7 (in particular paragraph 7) impose controls regardless of the measures taken by the service provider.
With regard to orders for the detection of grooming of children, the significant risk referred to in point (a) of the first subparagraph of paragraph 4 is deemed to exist if the following conditions are met
(a) the provider is a provider of interpersonal communication services
(b) it is probable that, despite any mitigating measures which the provider may have taken or will take, the service is being used to an appreciable extent for the solicitation of children
(c) there is evidence that the service, or another comparable service if one is not yet offered in the Union at the date of the request to issue a detection order, has been used in the last 12 months and to an appreciable extent for solicitation of children.
Conclusions
The situation is absolutely complex and delicate, and the risk of slipping into sterile polemic is very high; the suggestion after this article is to take the time to read the 89 articles of the standard and reflect on the enormous legal challenge behind the problem.
