In an era defined by rapid advances in artificial intelligence, balancing innovation with safety and responsibility has never been more critical. We sat down with João Galego, Head of AI at Critical Software, to explore his journey from early GPU-powered physics simulations to bleeding edge AI solutions.
In this interview, João shares his perspectives on the evolving landscape of AI regulation—from the EU’s phased rollout of the AI Act to the cultural tensions between Silicon Valley’s “move fast” ethos and Europe’s risk-averse mindset—and delves into the technical and organizational challenges of deploying AI in a safe way for users.
— Interview by Sandra Leonor
Q1. Can you briefly share your journey into AI and your current role at Critical Software?
A. I first encountered machine learning just after finishing university (2008–2013). I dove in by taking Andrew Ng’s Coursera course on Machine Learning, filling notebooks with notes—and that really sparked interest in me. Interestingly, I actually worked with GPUs as early as 2009 during a computational physics course, running lattice quantum chromodynamics (QCD) simulations.
After graduation, I cut my teeth in traditional software roles—quality and reliability testing, DevOps automation—and came back to AI when I sensed the limits of rule-based automation. I even started (but didn’t finish) a PhD in cognitive science, exploring brain–machine interfaces.
After several years at Siemens, I switched teams and started to explore Machine Learning applications in banking and finance. Then I moved to AWS to broaden my scope, but the opposite happened. ChatGPT’s launch in December 2022 was a turning point: in a couple of months, AI went from being a ‘side gig’ to a full-time job. At AWS I had the opportunity to talk to AI companies and understand what they were doing, which led to develop an even bigger curiosity for this technology and its potential.
Today at Critical Software, I’m building AI systems from the ground up—bringing together data scientists, ML engineers, and seasoned software engineers to apply robust engineering practices to AI.
Q2. How do you view the role of regulation in the AI ecosystem?
A. Regulation is essential. It forces all players—especially “move fast, break things” startups—to adopt safe, responsible practices. Without it, companies might misuse consumer data or overlook bias in their models. It took ChatGPT 5 days to reach 1M users and AI applications are now used by billions of users around the world. This comes with its own risks. Most users are not well informed on the consequences of third parties using their personal data and don’t know what mechanisms they have at their disposal to protect them from bad use of their data. Synthetic data generation has never been easier, which leads to a higher risk of. Good regulation sets high quality and safety standards without prescribing every technical detail; it’s a framework that we can refine over time.
Q3. Should AI be regulated like defense or healthcare technologies?
A. Yes and no. Any regulation must recognize AI’s probabilistic nature and other risks. We need assurances not only on model behavior but also on data provenance and representativeness. While critical domains like medical, energy or defense play by stricter rules, the same foundational principles—transparency, robustness, privacy—apply everywhere. Regulations should evolve through public consultation. The phased rollout of the EU AI Act is a proof of flexibility from the legislators, to test and iterate on the initial legislation, and to understand what makes sense and doesn’t, A/B testing regulation. Paraphrasing a paper I read recently from Helsing, AI is in a similar position today as to when software emerged and we started using existing electronics approaches, we understood it didn’t work and had to come up with new ones.
Q4. Should regulation be proactive (anticipating problems) or reactive (responding to incidents)?
A. Ideally proactive, with constant feedback loops. We should try to anticipate problems, but of course we can’t anticipate every scenario. Black swans are everywhere.
At Critical, we strive for proactive rules—defining safe operating scenarios and building “safety cages” around ML components—but we must also respond to unforeseen failures. The safety cage architecture includes a ‘checker’ layer that will safely shut down and/or isolate an AI module if it starts behaving outside its operational parameters. This decouples the AI model from the rest of the system and minimizes the risk of failure.
Q5. Do U.S. AI companies overstate the burden of regulation?
A. I get that OpenAI wants as much data access as possible. Their models are data hungry. But we can’t just give them a free pass to data, for the risks I’ve mentioned before. Yet the old saying “scarcity breads innovation” can be true: look at DeepSeek, which built powerful models despite U.S. embargoes on high-end GPUs. They created technical innovations to overcome the challenges they were facing.
I’m not saying that there should be tight regulation. We need a balance: allow experimentation under controlled rules, with rapid feedback loops to adjust regulation as technology and business needs evolve. There should exist a mechanism that ensures a constant and faster communication to question regulation, adjust it, change it or remove it.
Q6. Is this tension particularly strong in Europe?
A. Culturally, yes. The saying “U.S. innovates, Europe regulates, China imitates” used to hold true, but China is catching up fast, as we can see with DeepSeek, and Europe is slowly changing how it sees regulation. In the U.S., risk is part of the ethos; in Europe—and especially in Portugal—there’s more risk aversion. In the USA, failure is a war story, from someone who moved fast and broke a lot of things. In Europe, we exercise caution in every action. We need to learn from each other.
Q7. What are some risks that come from using this technology in products? Do you believe regulation can diminish them?
A. Most regulation is prescriptive about what you must guarantee—data anonymization, resilience against attacks, traceability—but not how to implement it. You must think from day one about where data is stored, how it’s transformed, and how to stop/interrupt bias and prevent leaks.
There are several risks: deepfakes, synthetic-data loops, and models “optimized for engagement”, trying to imitate the human behavior, can amplify misinformation. And I don’t think we’re ready to identify synthetic and fake information, and many times people don’t even care.
There’s a big risk that we’re training models with data generated from those very own models that would diminish their quality and be used to perpetuate biases present in the data used to train these models. The saying “garbage in garbage out” used for training is also true when people are interacting with it, and testing in the real world.
Q8. Which AI trends excite you most over the next 3–5 years?
A. Broad adoption in under-served domains like healthcare and aviation. I work in a privileged position where I can interact with different verticals and it’s clear that they operate at different speeds. Many promises (early cancer detection, melanoma screening) aren’t yet widespread because reliability and regulatory frameworks still need maturing. I’m also keen on human-in-the-loop systems that elevate, not replace, human decision-making, and I believe we’re going to see more of those in the future.