In 2023, when Replika users lost their AI companions due to software updates, it was not a minor software issue for them, but the loss of a “wife,” a “best friend,” or “a better sexting partner”. A similar situation occurred in 2025, when the AI companion Dot shut down, prompting users to hold digital funerals, sharing final screenshots and farewell messages as if real people had died.

If an AI companion is becoming so psychologically significant to people, any interference with the human-AI relationship may affect the user’s emotional stability, mental well-being, or psychological integrity, thereby violating the rights protected by the Convention. One obvious example of such a violation is a violation of the rights declared in Article 8, which protects everyone’s private and family life and safeguards the right to establish and develop relationships with the outside world, as was mentioned by the European Court of Human Rights (‘the Court’) in Bensaid v. United Kingdom. Continuing this line of reasoning, the Court has held that mental health should be regarded as a crucial part of private life, associated with moral integrity, and protected under Article 8 of the Convention.

Due to lack of regulation, users of AI companions face uncontrolled third-party intervention in the human-AI relationship, potentially violating the principle of personal autonomy. As owners of AI companions, private companies can alter, suspend, delete, or redesign the human-AI relationship without any user’s consent and without legal consequences. Such uncontrolled company actions, in the absence of proper governmental control and regulation, can affect people’s moral integrity, psychological health, and the private life rights protected by the Convention.

Being obliged to protect human rights and freedoms enshrined in the Convention, countries have not only obligations to prevent government violations but also positive obligations to adopt measures to secure human rights, as recognised in cases such as X and Y v. The Netherlands. That is why the lack of governmental regulations governing human-AI relations, aimed at protecting the Convention’s standards and principles in such cases, also constitutes a human rights violation.

On the one hand, adopting instruments such as the Framework Convention and the EU AI Act gives rise to hope for human rights protection in Europe. But on the other hand, we should be realistic: we are only at the start of the way, and the one thing countries can do now is react quickly and sometimes be more flexible and less bureaucratic to address the challenges of these new times.

The lack of specific laws on AI companions in the EU, as well as the absence of a comprehensive set of regulations governing human-AI relations and protections, will likely prompt the Court and local courts in Europe to issue rulings without waiting for governmental regulations.

Very soon, we will face a growing number of the above-mentioned issues arising from the need to expand the scope of human rights protection in human-AI relations. Notably, while human-AI relationships may pose emotional risks and negative consequences for the person, they also play a very positive role, helping lonely and elderly people. That is why, in its early cases involving human-AI relations, the Court should be very careful in interpreting the Convention and in determining who is violating rights and freedoms and how. In some cases, the harm may be caused by AI companions, but in others by governments or by providers of AI companions.

The era of human-AI relations has only just begun and will bring many surprises. Human-AI relations are no longer solely a matter of software use or service provision but fall within the scope of rights and freedoms already protected by the Convention, so states should no longer avoid their obligations.

There are projects in some jurisdictions – including China, Brazil, and Argentina – as to how these technologies could be employed in the judicial decision-making process to reduce case backlogs and enhance the delivery of justice. Contrarily, some countries – such as the UK, Ireland, New Zealand, and Canada – have taken an opposing stance to prevent the adoption of LLMs in core judicial functions, including determination and adjudication. Instead, they permit LLMs as support tools for judges and court staff to perform routine work like summarising text, speech writing, searching for information, sources, and precedents, enhancing clarity and linguistic flourish, translating, and transcribing.

However, such ‘innocent’ use of LLM tools in routine or secondary court functions has the potential to influence judicial cognition and thereby affect the core judicial decision-making process. Institutional permission and repetitive use of these tools may lead to several cognitive effects, such as automation bias, cognitive offloading, anchoring, and the framing effect. Automation bias leads to over-reliance on automated recommendations, while cognitive offloading reduces mental effort through external tools. Anchoring causes over-dependence on initial information, and the framing effect shows how decision-making shifts based on information presentation.

Judging involves reading the case materials, identifying the issues, and developing the reasons by themselves, whereas a preliminary recommendation produced by an LLM system has the potential to serve as a starting point for such cognitive involvement of judges, anchoring the point of departure. On the other hand, when a judge outsources the meaning of legal lexicon, a summary of long texts, or the identification of precedents to an LLM, they ultimately engage in cognitive offloading. This cognitive offloading can lead to a superficial review process where judges make only ‘surface-level’ scrutiny or modifications to LLM-generated text, bypassing the deep cognitive work required to fully understand the legal reasoning and verdict.

Therefore, simply relying on a human judge to use LLM systems in preparatory stages, or to audit the veracity of the outcomes of such systems subsequently in the judicial decision-making process, is potentially problematic. Though judges are trained to be impartial, cognitive and behavioural science demonstrates that no decision-maker is immune to systematic cognitive biases. Decades of behavioural research demonstrate that professional expertise does not eliminate susceptibility to anchoring, framing, or automation bias. Judges, like other professionals, work within cognitive constraints.

These cognitive defects are significant because human judges are regarded as the principal safeguard to check and control any type of bias and are also regarded as neutral decision-makers to ensure justice. In the circumstances of heavy judicial workloads, exhaustion, and in some cases institutional pressure to deliver timely decisions with limited resources, cognitive issues can arise. As a result, it can undermine the fundamental principles of fair trial, including judicial independence, the judicial duty to provide well-reasoned decisions, and ultimately the rule of law.

The inherent persuasive features of LLMs that randomly generate text with human-like reasoning and confidence, in spite of factual or legal accuracy, may make human judges vulnerable to this uncritical adoption of LLMs even as a support tool for judicial decision-making. This creates a deep tension with the fundamental and uncompressible judicial duty of independent reasoning, signifying that without precise safeguards and a deep understanding of this cognitive susceptibility, the integration of LLMs in adjudication, even as secondary tools, could unintentionally undermine the deliberative process it seeks to augment. Hence, in an increasingly technology-driven judiciary, it is critical to examine whether judges’ cognitive limitations prevent them from effectively guarding against aggravating this tension.

Due diligence in international human rights law refers to the proactive obligation of states to take reasonable steps to prevent, investigate, punish, and remedy human rights violations – including those arising from the state’s own systems or through third parties acting under its authority. In the context of government-deployed AI, this encompasses identifying foreseeable rights risks prior to deployment, building institutional capacity to monitor algorithmic outcomes, and ensuring accessible mechanisms for remedy. The concept originates in Velásquez Rodríguez v Honduras (IACtHR, 1988) and has since been developed through subsequent UN treaty body jurisprudence and scholarly development.

Under international human rights law, states have a duty to respect, protect and fulfil the rights enshrined in treaties such as the International Covenant on Civil and Political Rights (ICCPR) and the International Covenant on Economic, Social and Cultural Rights (ICESCR). These obligations are not passive: they require states to take reasonable measures to prevent foreseeable harm to rights holders [8]. This analysis draws in particular on three soft law instruments: the UN Guiding Principles on Business and Human Rights (2011); the UNESCO Recommendation on the Ethics of Artificial Intelligence (2021); and the Council of Europe Framework Convention on Artificial Intelligence and Human Rights (2024). While not binding per se, these instruments provide authoritative interpretive guidance on how existing treaty obligations apply in algorithmic governance contexts.

Under established international human rights law, states bear an affirmative obligation to assess human rights impacts of their actions and adopt measures to address risks. These obligations persist regardless of whether the state develops the technology itself or relies on private vendors. Responsibility cannot be deferred on the basis of third-party involvement – a robust due diligence framework must be in place. More specifically, the obligations derive from: (1) Article 2(1) ICCPR – the general obligation to respect and ensure rights without discrimination; (2) Article 2(1) ICESCR – the obligation to take progressive steps toward full realization of rights; and (3) Principle 1 of the UN Guiding Principles on Business and Human Rights, which affirms that states must protect against human rights abuses by third parties, including private vendors, operating within their jurisdiction.

AI systems pose distinctive challenges for due diligence precisely because they are often non-transparent and adaptive. Traditional administrative instruments can be evaluated against statutory criteria. AI models, by contrast, can change behaviour based on data inputs. These features do not exempt states from their duties; on the contrary, the complexity of the system increases the diligence required. If a state deploys an algorithmic eligibility tool without the capacity to assess how it functions, it places itself at risk of violating core human rights obligations.

Consider the obligation to guard against discrimination. Article 2(1) of the ICCPR requires states to ensure that rights are recognized and respected without distinction of any kind. If a government uses an automated risk assessment tool that systematically biases outcomes against a protected group — and lacks the institutional capacity to analyse or mitigate that bias — it becomes difficult to argue that the state has taken reasonable measures to prevent discriminatory outcomes. The foreseeability of such harm is increasingly documented in academic literature and technical studies, yet states lacking technical oversight networks remain ill-equipped to meet their due diligence obligations.

Similarly, the right to effective remedy — a central component of international human rights law — imposes procedural obligations. Article 2(3) of the ICCPR requires states to ensure that anyone whose rights are violated has access to an effective remedy through competent judicial, administrative or legislative mechanisms. When an AI system produces an adverse decision, due diligence requires that there be mechanisms to challenge that decision and correct errors. Without institutional capacity to audit, review and correct algorithmic outputs, effective remedies remain theoretical.

Crucially, the standard of due diligence is measured by reasonableness, not convenience. States are judged against the backdrop of what is feasible to prevent foreseeable human rights harms. Where a technology is integral to public administration and its risks to human rights are documented, failing to establish oversight mechanisms, transparency requirements or accessible grievance processes raises serious questions under the state’s existing obligations. The fact that AI systems are complex does not diminish the obligation — if anything, it heightens the expectation that states will invest in capacity to govern their use responsibly.

From an independent legal analysis perspective, due diligence provides a familiar legal lens through which to assess the human rights implications of government-deployed AI systems. While soft law instruments do not themselves create new rights, they interpret existing obligations in contemporary settings and are relevant to assessing due diligence. States that deploy AI without such frameworks to monitor and mitigate rights impacts risk falling short of their binding international duties.

Dialect, Language and Nationality

The foundational flaw in using speech analysis – relying on accents and linguistic patterns – lies in the shared linguistic heritage of the Indian subcontinent.  The nature of the data used to train such a system remains unclear but it may include publicly available speech datasets, government-held audio records, and field-level collections. State authorities have previously misidentified Indian citizens as ‘foreigners’, leading to wrongful deportations.. An AI tool relying on Bengali dialects is inherently error-prone, as linguistic boundaries between West Bengal, Assam, and Bangladesh significantly overlap. Experts have shown concern noting that the Bengali language has regional variations and no unique markers to distinguish the language spoken in states like West Bengal, Assam and other parts of India from Bangladesh.

Historical context further underscores the problem. Following the Partition of India and the creation of Bangladesh in 1971, a consequence of the Bangladesh Liberation War, millions of people migrated across borders. Hence, language and its particular dialects remain rooted in geography, and not nationality or religion. For instance, the Bengali-speaking population in Assam’s Barak Valley speak Sylheti, a dialect spoken in Bangladesh’s Sylhet district. Any AI system attempting to infer citizenship from such speech patterns is likely to be structurally incapable of reliable accuracy without an extraordinarily large dataset.

AI Opacity and Risks of Violation of Fair Procedure

In Selvi v. State of Karnataka, the Supreme Court of India recognised that Article 21 included substantive due process rights.. In  Maneka Gandhi v. UOI, the Court held that any procedure affecting life or liberty must be fair, just, and reasonable. Thus, individuals faced by state decisions must be able to understand, challenge and have access to evidence and reasoning relied upon against them.

Most AI models predict through deep neural networks, operating as ‘black box models’ where only the input and the output is visible to users while the reasoning process remains opaque. Currently, most AI platforms run on the black box model and the probability of implementing such non-transparent systems is high in this case. This lack of explainability undermines procedural fairness, as affected individuals cannot understand, contest, or rebut the basis of adverse decisions.

India’s recent experience with algorithmic tools illustrates these dangers. In the investigations relating to the North-East Delhi riots, the Delhi Police used facial recognition tools like ‘Amped FIVE’ and ‘AI Vision’ without disclosing methodology or accuracy rates.  Matches above 80% were treated as positive identifications. However, such thresholds can be problematic. For instance, as the American Civil Liberties Union pointed out, a similar threshold applied in Amazon’s Rekognition system resulted in significant false identifications.

These concerns are magnified in sensitive contexts such as immigration enforcement when ‘error rate’ is measured in human lives, including refoulement, family separation, and statelessness. Such outcomes undermine India’s commitment towards IHRL and United Nation’s AI Resolution to promote transparent and explainable AI systems.

Privacy

While K.S. Puttaswamy v. Union of India recognised privacy as a fundamental right, India’s legislative framework offers limited protection against AI-driven profiling in immigration enforcement. The Digital Personal Data Protection Act, 2023 lacks specific safeguards for sensitive data such as linguistic biometrics. Broad exemptions under Sections 7(c) and 17(2)(a) permit state processing in the name of sovereignty. Further, Section 44(3) weakens transparency by permitting the government to withhold information involving personal data, effectively insulating opaque AI profiling systems from public and legal scrutiny.

Conclusion

The use of AI to determine citizenship raises profound concerns for due process and human dignity. While the European Union has recognised a right to explanation for AI systems, India has no comparable statutory recourse. In a context where dialects cut across borders and communities, algorithmic classification is structurally unreliable, rendering individuals without recourse.

As the Bombay High Court observed in TPL HGIEPL Joint Venture v. Union of India, ‘the principles of natural justice and fairness are too valuable to be sacrificed at the altar of AI expediency.’ Deploying opaque and error-prone systems in citizenship determinations directly infringes Article 21, which guarantees life, personal liberty, and procedural fairness.

Across Europe, public authorities are exploring data-driven tools to support decisions in child protection and youth justice systems. These systems promise consistency and efficiency, identifying patterns of potential neglect or harm through statistical modelling. Yet predictive technologies operate on historical datasets. Where those datasets reflect socio-economic inequalities, over-surveillance of certain communities, or institutional bias, algorithmic outputs may reproduce and amplify existing disparities rather than mitigate them. Although the Dutch childcare benefits scandal did not arise directly within child protection law, it exposed a structural danger increasingly relevant across Europe: when algorithmic systems classify families through predictive models, the consequences may profoundly affect children’s rights and family integrity.

This tension strikes at the core of Article 3 of the UN Convention on the Rights of the Child (CRC), which requires that the best interests of the child be a primary consideration in all actions concerning children. As clarified by the UN Committee on the Rights of the Child in its General Comment No. 14 (2013), this principle requires an individualised assessment grounded in the specific circumstances of each child, taking into account their unique situation, needs, and vulnerabilities.

Similarly, decisions that interfere with family life engage Article 8 of the European Convention on Human Rights (ECHR), and are only justified where legal, necessary, and proportionate to the legitimate aims pursued by the interference. When algorithmic risk scores significantly influence decisions about monitoring, intervention, or potential removal, transparency and individualised judicial oversight become essential safeguards.

The Dutch childcare benefits scandal illustrates how reliance on automated risk categorisation can undermine procedural fairness. Affected families often struggled to understand how decisions were reached or to effectively challenge risk classifications, as documented in official investigations and public reporting on the scandal. In child protection contexts, similar opacity could weaken trust in institutions and compromise the legitimacy of state intervention.

This does not mean that technology has no role in safeguarding children. Data analysis may help identify systemic gaps or support early intervention strategies. As illustrated by the Dutch childcare benefits scandal, automated risk assessments may flag families as “high risk” without adequately accounting for individual circumstances, leading to disproportionate and unjust outcomes. However, algorithmic tools must remain subordinate to human rights principles. The best interests of the child require more than statistical probability; they demand contextual sensitivity, proportionality, and meaningful judicial oversight.

The broader European regulatory landscape reinforces this concern. The emerging framework of the European Union Artificial Intelligence Act recognises that certain AI systems used in public administration and law enforcement pose “high-risk” implications for fundamental rights. Where such systems affect children and families, heightened scrutiny is not optional but necessary. Algorithmic efficiency cannot replace individualised human judgment grounded in dignity and fairness.

As Europe moves toward greater reliance on AI-driven governance, child protection and youth justice systems must resist technocratic overconfidence. Algorithms may inform decisions but they cannot determine them. A rights-based approach requires that predictive tools operate within clear normative boundaries, ensuring that the best interests of the child remain a living legal principle rather than a statistical abstraction. If algorithmic governance expands without robust, rights-based safeguards, the promise of child protection may quietly transform into a new architecture of digital suspicion.

Posidex Technologies Private Limited built Samagra Vedika for the Telangana state government. The system pulls citizen data from over 30 government databases and creates “360-degree profiles.” Machine learning algorithms then compare records across these databases through a process called “entity resolution” to verify welfare eligibility and flag suspected fraud. Officials must consult the system before approving any welfare application. They retain the formal authority to override algorithmic outputs. In practice, they defer to the machine. An Amnesty International technical investigation published in April 2024 confirmed that the system raises serious human rights concerns. Neither Posidex nor the Telangana government has released any source code or accuracy data.

Article 11 of the ICESCR protects the right to adequate food. India’s National Food Security Act 2013 guarantees subsidised food grains to roughly two-thirds of the population. The Supreme Court of India, in PUCL v Union of India (2001), recognised the right to food as part of the right to life under Article 21 of the Constitution. An opaque algorithm now strips citizens of this entitlement. The state offers no explanation and no notice. Affected individuals receive no meaningful opportunity to contest the decision. Instead, excluded beneficiaries carry the burden of proving the algorithm wrong. This reverses the constitutional duty the state owes its citizens.

Other countries have confronted similar failures. In the Netherlands, the Hague District Court struck down a welfare fraud algorithm in the SyRI case (2020) under Article 8 of the ECHR, holding the system’s opacity violated proportionality. In Australia, courts ruled the Robodebt automated debt recovery scheme unlawful. The government repaid AUD 1.8 billion to over 400,000 people. In Serbia, Amnesty International’s 2023 research showed algorithmic welfare decisions disproportionately excluded Roma communities and persons with disabilities from social assistance. Each case carries the same lesson: opaque algorithms making high-stakes decisions about vulnerable populations demand transparency, oversight, and enforceable remedies.

India has no legal framework for algorithmic accountability in welfare delivery. The Digital Personal Data Protection Act 2023 does not address algorithmic discrimination. The law does not mandate impact assessments. Section 17 exempts government agencies on grounds of national security and public order. Welfare recipients have no statutory route to challenge algorithmic exclusions. Meanwhile, the Samagra Vedika model spreads. Haryana, Maharashtra, Tamil Nadu, and Uttar Pradesh have adopted or are developing similar platforms. In Haryana, a pension algorithm declared thousands of living pensioners dead, terminating their old-age and widow pensions. These individuals, many elderly and illiterate, were forced to prove they were alive.

Three reforms are urgent. First, India needs legislation requiring algorithmic impact assessments before any automated system enters welfare delivery. Second, the government must guarantee affected individuals a right to explanation and a meaningful process to challenge algorithmic decisions, consistent with Article 14 of the ICCPR. Third, private developers of welfare algorithms must face independent audits and transparency obligations. The Council of Europe’s Framework Convention on AI (2024) and the EU AI Act’s classification of welfare-related AI as “high-risk” provide working models. Without these safeguards, India’s expanding digital welfare infrastructure will deny food, pensions, and housing to the people who need them most.

The Limits of Publicity Rights

Deepfakes are AI-altered audio, visual, or audio-visual content depicting individuals in a highly realistic manner, generated through deep learning. They include image reconstructions, voice cloning, and pornographic material. As publicity rights protect the commercial appropriation of one’s persona, including name, image, likenesses, and voice, celebrities such as Jackie Shroff, Arijit Singh, Aishwarya Rai, Anil Kapoor, have secured injunctions against platforms and creators for blocking deepfake content. Yet, publicity rights can be an inappropriate avenue to address deepfake harm.

Publicity rights in India lack a coherent doctrinal rationale, with courts frequently conflating the tort of passing off with the constitutional right to privacy. Stemming from the former, publicity rights are commercial in nature and penalise deceptive misrepresentations in trade based on a finding of goodwill; confusion; and resulting damage. Indian courts, however, have repeatedly presumed confusion merely based on the identifiability of a celebrity, without assessing whether consumers were indeed misled. This doctrinally flawed approach, coupled with expansive remedies like dynamic, even ex-parte injunctions, effectively grants celebrities an excessive monopoly over image-related use even when harmless or expressive, thus chilling free speech and trade. For instance, in Mohan Babu, an ex-parte injunction restrained even subscribers of YouTube channels from using the name/image of the actor.

As ex-post facto remedies that don’t prevent deepfake creation, publicity rights actions only help celebrities who can afford expensive litigation, leaving ordinary people vulnerable. Focused on the celebrity’s commercial interests, they also do not address wider misinformation harms that may endanger public safety, such as in the Ankur Warikoo case, where deepfakes offered falsified financial advice, risking serious consumer harm.

Deepfakes involve malicious AI-generated impersonation resulting in privacy, autonomy and dignity harms, which affect vulnerable groups disproportionately. Female celebrities including Rashmika Mandanna, Alia Bhatt, and Katrina Kaif have been subjected to non-consensual sexual deepfakes, a form of technology-facilitated gender-based violence. Publicity rights do not capture these harms that directly threaten the constitutional guarantees under Article 21, as affirmed in Puttaswamy. While there may be an overlap, relying on publicity rights to address deepfakes risks further doctrinal confusion and could suppress legitimate Gen-AI uses.

MeitY’s Platform-Governance Amendments

The proposed amendments mark India’s first step towards deepfake regulation at intermediary level. However, they risk undermining lawful expression.

First, the requirement for platforms enabling the creation of  synthetically generated information (SGI) to label such content with permanent metadata is grounded in an expansive definition. The proposed Rule 2(1)(wa) defines SGI as: ‘artificially or algorithmically created, generated, modified or altered using a computer resource, in a manner that such information reasonably appears to be authentic or true’. While promoting transparency, the definition prioritises technical modification over deceptive intent, sweeping within its ambit even harmless alterations like photo filters, memes, cinematic enhancements, and scientific simulations. Echoing the overreach seen in publicity rights enforcement, this threatens overregulation of creative expressions. By contrast, the EU’s Artificial Intelligence Act focuses on deceptive potential, offering a narrower and more targeted model.

Second, the rule’s effectiveness also depends on whether Gen-AI models qualify as ‘intermediaries’ under the Information Technology Act, 2000 (IT Act). As such an intermediary typically performs a passive role limited to hosting or enabling transmission, rather than content-creation, few AI models would meet the mark, limiting enforceability. This imposes compliance burdens on intermediaries while leaving unregulated model developers, encouraging over-enforcement by platforms without addressing deepfake generation at its source.

Third, the proposed amendments risk undermining the safe harbour principle under Section 79 of the IT Act, which grants intermediaries immunity from liability based on their inability to screen massive volumes of user-generated content. Encouraging proactive content-policing while retaining the immunity fundamentally contradicts the rationale for safe harbour and might incentivise excessive moderation and automated filtering, disproportionately impacting free expression.

The proposed amendments represent a mid-layer regulatory intervention into India’s deepfake crisis,  placing primary obligations on intermediaries rather than upstream developers or trainers who design and deploy technologies capable of deceptive synthetic data generation or the downstream malicious actors who generate and circulate deepfakes. While potentially reducing reliance on expensive litigation, this is likely to encourage precautionary moderation practices resulting in over-censorship and chilling effects on legitimate expression by online creators and speakers, impacting political spoofs, parody, and satire.

The regulatory debate on AI in the context of human genetics and genomics involves two continuums — risks and innovation — leading to contentious discussions over global, local, and self-regulation. Given the potential benefits of AI, companies and even countries are investing in and promoting the development of AI. It is encouraging that companies are volunteering to develop ethical frameworks for AI regulation; however, they often miss the core values human rights protect. It triggers an inquiry into why global governance of AI to human genetics and genomics is crucial, and how the human rights framework can provide effective global governance.

Human genomics involves the increased application of data-driven science, where the application of AI is manifold, ranging from diagnosis and screening to drug development and treatment. It holds great promise in healthcare by adding precision, speed, and optimisation to genetic diagnosis, precision medicine, and precision health. It can help promote the next wave of medical and healthcare immensely.

However, it can also deepen and enlarge human rights concerns, such as data privacy and biases in medical algorithms and decision-making. Diversity in genetic datasets is important for creating algorithms used to train machine learning models. The lack of diversity in global genetic datasets and the exclusion of marginalised populations in national genetic datasets may lead to discrimination and inequity in genetic research and genomic medicine. Thus, research findings and treatment using these datasets may be inaccurate and less useful for the excluded populations. It may also create a new subclass, privileged with the genetically advanced treatment, leaving others from its loop and deepening health disparities.

Furthermore, the black-box problem (that is, the difficulty in comprehending how a complex system arrives at a decision) complicates the explainability of AI-driven decision-making in the field of human genetics and genomics. Although existing ethical, legal, and policy frameworks address developments in the domain of human genetics and genomics, notably the Universal Declaration on the Human Genome and Human Rights (1997) and the Declaration of 2002, followed by regional human rights instruments and specific legislations of various countries. However, new and unprecedented challenges arising from the AI-human genetic and genomic interface warrant a specific approach. At the intersection of AI and genetics, robust data anonymisation is extremely difficult, and AI’s surveillance, data tracking, and pattern recognition capabilities exacerbate human rights risks.

The relevance of a human rights gaze into the regulation and governance of AI-human genetics/genomics is pertinent on several counts. The universality of human rights appears to be compatible with the universal (cross-border) nature and global impact of AI and genetics. The human rights approach takes a human-centric perspective, whereas technology-driven policies guide ethical frameworks that are developed and implemented by companies with profit as the primary focus. While ethical frameworks are open to varied interpretation, human rights are more established and uniform in their application. Finally, contrary to human rights, which are enforceable and actionable, ethical frameworks appear weak due to their reliance on self-regulation and principles that can be ambiguous and unclear.

The right to science, which has been recognised as a cultural right in international human rights instruments, could help regulate AI and genetics equitably, as it ensures the benefits of science are available to all. In the genetic and genomic research, data stewardship and data security also require inspiration from the HR framework. Scientists need to be more committed to respecting the human rights of individuals and groups who participate in their experiments, whether willingly or not.

Given the serious impact of AI on humanity (such as autonomous weapon systems, and genetic discrimination), human rights norms become essential and should therefore support the global governance of AI. AI complements humans; therefore, a debate about their harmonious and respectful coexistence makes sense. Human rights-centric AI regulation has a blend of coexistence, managing vast genomic datasets, harmonising and standardising this information, and employing machine learning techniques to clean missing data and sort human genomic data.

A Test Case: xAI in the U.S.

Reflections on the evolution of this field can be drawn from the case of xAI, a company dedicated to the development of AI, which, since mid-2024, has operated a data center near Boxtown, Memphis.

Some researchers have found that the presence of pollutants in the air, such as nitrogen dioxide, has increased significantly compared to the pre-xAI period in areas surrounding the data center and, to some extent, in nearby zones. This increase has been linked, with a high level of probability, to the operation of the new xAI data center.

On 17 June 2025, the Southern Environmental Law Center, an environmental advocacy organization in the United States, sent a formal notice to xAI on behalf of the NAACP, the National Association for the Advancement of Colored People, demanding that the company cease operating polluting gas turbines. The notice argued that xAI’s decision “to install and operate dozens of polluting gas turbines without any permits or public oversight is a clear violation of the Clean Air Act”. Boxtown is, in fact, a community predominantly composed of Black residents, originally founded by individuals who were formerly enslaved.

At the local level, the Shelby County Health Department, the public authority responsible for safeguarding community health and environmental standards in the Memphis area, implemented the installation of air monitoring instruments to assess the potential impact of the data center. Similar initiatives were also announced by Memphis Community Against Pollution, which is seeking independent monitoring mechanisms to ensure transparency and accountability.

Yet, despite these measures, residents have continued to raise concerns. Communities in Boxtown and the surrounding areas have reported a range of health issues, from respiratory discomfort to worsening asthma, suggesting a potential link between the operation of xAI and declining health conditions. The World Health Organization has associated similar increases in pollutant levels with the types of symptoms reported by these communities.

The Role of the UN Guiding Principles in the Protection of the Environment and Health

This case is situated within a broader challenge: balancing the rapid expansion of AI training with the protection of the rights to a healthy environment and to health as suggested by the UN Guiding Principles on Business and Human Rights (UNGPs).

According to Principle 13, business enterprises have a responsibility to “avoid causing or contributing to adverse human rights impacts through their own activities, and address such impacts when they occur”. Principle 17 further explains that human rights due diligence should be initiated as early as possible in the development of a new activity, such as the construction and operation of an AI data center, including its management of gas emissions. Moreover, Principle 18 states that this process must involve “meaningful consultation with potentially affected groups”. Applied to the case described above, this would require identifying from the outset which communities may be affected – such as the residents of Boxtown – and taking into account the distinct risks to which they are exposed, including not only environmental damage but also health-related harm.

Finally, Principle 22 emphasizes that “where business enterprises identify that they have caused or contributed to adverse impacts, they should provide for or cooperate in their remediation through legitimate processes”.

Although these principles are not legally binding, they outline a clear framework for action that companies should follow. Since AI training and the operation of data centers are associated with increased greenhouse gas emissions, as well as significant water and energy consumption, establishing a link between AI-related environmental pollution and adverse human rights impacts – whether in the United States or elsewhere – would represent a significant step forward in protecting the human right to a clean and healthy environment, and in preventing future environmental harm. This can contribute to enhancing the protection of human rights.

What is interesting to note is the fact that these techniques have been subjected to a complete prohibition, and have not been made contingent on the consent of the data subject. The only exception, under Recital 29, applies to the use of medical techniques deployed with explicit consent, e.g. the use of Deep Brain Stimulation to cure neurological disorders. Such a blanket prohibition differs, for instance, from the framework for processing of special categories of personal data under Article 9 of the General Data Protection Regulation (GDPR), which can still be processed with informed consent. Assuming that the use of algorithmic systems envisaged under Article 5(1)(a) of the AI Act involves processing of data on cognitive and behavioural states of data subjects, it is apparent that the AI Act possibly carves out a special category of data derived from a person’s internal decision-making processes that is afforded even higher protection than that ordinarily available under Article 9 of the GDPR. 

The reasons for providing greater protection to data relating to a person’s cognitive and affective states appear to be obvious: as the Information Commissioner’s Office (ICO) in the UK notes (p. 19), consent in the realm of “neurotechnology” may be an inadequate safeguard when it comes to ensuring control over one’s personal data. In a normal situation, by providing consent to processing of personal data, the data subject retains conscious control over what items of personal data are being divulged. When neurotechnology itself manipulates and modifies the data subject’s cognitive and behavioural functioning by directly modulating electrical activity or artificially stimulating certain areas of the brain,  the “voluntary” and “informed” nature of consent becomes suspect. Further, reports have emerged of neurotechnological advancements that purport to have the ability to convert brain scans into transcribed speech in real time. In the face of such technology, even if initially consent is provided, one has no control over what kinds of data may be unwittingly disclosed to the outside world, let alone determine the inferences that may be revealed from them. 

From a rights perspective, the prohibition in Article 5(1)(a) is itself an expression of the unqualified protection that is afforded to unexpressed “thought” or the “forum internum” in the European Convention on Human Rights (ECHR). Under Article 9(1) of the ECHR, protection is granted to “freedom of thought, conscience and religion”. However, while Article 9(2) envisages limitations on manifestation of one’s religion or beliefs, the provision is silent on any possible limitations on the freedom of “thought”. One possible interpretation of this deliberate wording points to the fact that unexpressed “thought” or the forum internum under the ECHR is given an inviolate and absolute status, free from any potential limitations or restrictions. This can explain why under Article 5(1)(a) of the AI Act, technological advancements that tinker with “thought” at its most basic level, namely, the cognitive process of thinking, are not permitted even with consent, subject to the medical use exception. 

While technological advancements ought to be welcomed, it is imperative to ensure that the freedom to preserve one’s innermost thoughts and the autonomy to make choices are not tinkered with. Viewed from this angle, Article 5(1)(a) of the AI Act acts as a guardrail that effectively safeguards the inviolability of “thought” from neurotechnological intervention.