History Of Deepfakes

The term “deepfake” burst into public consciousness in late 2017 and originated from Reddit, where a user of the same name shared manipulated pornographic videos, often of celebrities, by superimposing their faces onto existing footage. This initial, notorious application combined “deep learning” algorithms with “fake” media, and the release of underlying open-source code made the creation process available to everyone, allowing individuals with moderate technical skills to produce their own versions.

Since then, the technology has evolved at a breakneck pace, moving from these early, often discernible, manipulations to increasingly sophisticated and convincing fakes capable of being used for political disinformation, financial scams, and even live impersonations, marking a swift and concerning progression from niche internet phenomenon to a mainstream societal challenge.

Case Studies: The Real-World Impact of AI Impersonation

High-profile cases have demonstrated the devastating potential of this technology. The following two case studies served as the source of inspiration that drove the research MWR CyberSec did into this subject.

Case Study 1: The Elon Musk Deepfake Scams in South Africa

In South Africa, a series of sophisticated deepfake scams emerged, leveraging the likeness of Elon Musk and other prominent local billionaires like Johann Rupert and Patrice Motsepe. Scammers created convincing videos where these personalities appeared to endorse AI-powered cryptocurrency trading platforms. These deepfakes promised outlandish returns – for instance, turning a R4,700 investment into R30,000 in a single day.

The deepfake videos were remarkably well-produced, with “Musk’s” voice even mimicking local accents to enhance credibility. These videos circulated widely on social media, with some attracting hundreds of thousands of views. The consequence was substantial financial loss for numerous investors. In one documented instance, an individual invested and lost R5 million to one such scheme. While precise overall numbers are hard to ascertain, the widespread nature of these campaigns suggests that over 150 individuals could have fallen victim to this one specific campaign.

The following two videos show how one such deepfake scheme was created using a real existing interview of Elon Musk on Wall Street Journal and lip syncing it to a different audio source.

This case highlights not only the commercial impact through direct financial losses but also the significant social impact, as it erodes public trust and preys on the familiarity and authority of well-known figures.

Case Study 2: Arup’s $25 Million Lesson in Live Deepfake Deception

In a chilling demonstration of how deepfakes can infiltrate corporate settings, a multinational engineering firm called Arup fell victim to a $25 million scam in 2024.

A finance employee based in the company’s Hong Kong office received an email, purportedly from the UK-based Chief Financial Officer (CFO), requesting urgent and confidential fund transfers.

Initially, the employee was sceptical about the email, however, he was then invited to a video conference call. On this call, attackers convincingly impersonating the CFO and other senior executives (whose likenesses and voices were deepfaked in real-time) assured the employee that the instructions were legitimate and allayed concerns. Convinced by what appeared to be a legitimate, multi-participant video meeting with trusted colleagues, the employee authorised transfers amounting to approximately $25 million (HKD 200 million) to accounts controlled by the fraudsters.

This incident was a stark wake-up call, proving that live, real-time deepfakes are now sophisticated enough to deceive professionals in a business environment. The attack resulted in massive financial loss and underscored the potential for severe reputational damage to organisations that fall prey to such schemes. It also highlighted the psychological manipulation involved, as the live video interaction effectively overrode the employee’s initial skepticism.

How These Things Actually Work: The Technology Behind the Deception

Deepfakes leverage sophisticated AI to superimpose existing images and videos onto source images or videos (for video deepfakes) or to synthesise a target person’s voice (for audio deepfakes). Here is a general overview of the process that goes into making a deepfake:

Phase 1: Data Collection

For both video and audio deepfakes, source material is required for the target person that will be deepfaked. Typically, a significant amount of high-quality video, images and audio is required of the target person in order to train AI models to perform the deepfake.

To perform training, processing has to be performed on the source data such as transcribing audio, cropping images, feature extraction, and various other actions that enhance the training process.

Phase 2: Model Training (Teaching the AI)

This is the most computationally intensive phase of the process. The source material gathered by the attacker is fed to a model and training is performed. For video content, this involves learning the unique facial features, expressions, and nuances that make up the target person. For audio training, it involves learning the characteristics of the target’s voice, including pitch, timbre, intonation and rhythm.

Phase 3: Generation & Refinement (Creating the Fake)

Once the model has been trained, a deepfake can be produced. This can take the form of pre-recorded video or a live performance deepfake. The trained model from phase 2 generates the target person’s face or voice based on the input it receives from either a driving video or webcam and microphone.

For pre-recorded content, post processing and refinement could also be performed to make the deepfake look and sound more realistic. Lip syncing could be performed or visual artifacts might be edited or hidden with overlays.

Pre-trained models

Various pre-trained models are also available that can be used to produce deepfake content. These tools completely eliminate the need to perform vast source material gathering or training a model for millions of iterations, but rather allows a user to upload a short audio clip or even a single picture in order to start the deepfake process. The following video from research done by Bytedance shows just how advanced these pre-trained models can be.

It should be noted that the tool mentioned above was not publicly available at the time of writing due to ethical concerns by the developers (Good!). It does however demonstrate that with the rapid progression of AI tools in our modern age, the limitations for creating deepfakes are becoming less and less of a barrier to entry for attackers.

Technical Shortcomings Seen In The Practical Application Of Deepfakes

During MWR’s own attempts at recreating these techniques, some technical shortcomings were encountered that could help identify poorly made deepfakes.

Audio Deepfake Shortcomings

• Unnatural Cadence and Pace: AI models can struggle with rhythm. Listen for speech that is unnaturally fast or slow, as this can cause the AI to generate noticeable glitches or distortions.
• Volume Changes: Rapid volume changes from loud to quiet or vice versa can often lead to audio artifacting (that robotic sounding voice) being produced by the model.
• Whispers: Whispering lacks strong vocal cord vibration (pitch), which is a key feature that audio models rely on. Consequently, cloned whispers often sound distorted, breathy, or may have bizarre tonal inclinations.
• Context is King: The most powerful detection tool is your own familiarity with the person supposedly speaking. If you know them well, you may notice that their diction, tone, or emotional inflection is “off”. Trust your intuition if the voice sounds like them, but the way they are speaking doesn’t.
• Vocal Range Mismatch: Real-time voice changers are particularly vulnerable when there’s a significant difference between the input and target voices. For example, if someone with a naturally high-pitched voice attempts to clone a very deep, low-pitched voice in real-time, the output may sound strained, tinny, or unstable. This doesn’t help that much in detecting these as an attacker would likely pick a target that more closely resembles their own voice.

The audio samples below demonstrates some of the shortcomings:

Video Deepfake Shortcomings

• Masks: A deepfake model needs to constantly detect a face in the source video to overlay a target face onto it. If this detection is interrupted—perhaps by a hand passing in front of the face or gestures that the source video didn’t cover (think sticking your tongue out), or poor lighting, the “mask” can break. The results are often jarring and obvious, ranging from features being incorrectly mapped to the fake face momentarily vanishing altogether.
• Unnaturally Smooth Skin: The deepfake generation process often involves compressing and then reconstructing facial features. This can lead to a loss of fine detail. Look for skin that appears unnaturally smooth, almost like a digital airbrush has been applied. Details like pores, wrinkles, fine hairs, or even stubble may be smoothed over or absent entirely, giving the person a doll-like appearance.
• Irregular Gestures and Behaviour: This is another context-based clue. We all have unique mannerisms, head tilts, and hand gestures that accompany our speech. A deepfake may replicate a face perfectly, but if the gestures or expressions don’t match the person you know, it’s a major red flag. If a normally animated friend is suddenly stiff and inexpressive on a video call, or vice versa, it could indicate that you’re watching a digital puppet, not a real person.

The video below demonstrates some of these shortcomings in an exaggerated manner:

Mitigations and Staying Vigilant: What Can Be Done?

The U.S. Department of Homeland Security (DHS), in its report “Increasing Threats of Deepfake Identities” emphasises that the threat of deepfakes comes not just from the technology itself, but from our natural inclination to believe what we see and hear. Even less sophisticated deepfakes can be effective in spreading misinformation.

The DHS report outlines that there is no single, universal solution to the deepfake problem. Instead, a multi-pronged approach is necessary, encompassing the following phases of a deepfake attack:

1. Technological Innovation:

• Developing and improving deepfake detection technologies. This is an ongoing “cat and mouse” game as generation techniques become more advanced.
• Exploring digital watermarking or authentication technologies that can help verify the authenticity of media.
• This phase is largely dependent on developers and organisations that have to consider the ethical implications of what they are developing but also how these safety measures could be added.

2. Education and Awareness:

• Critical Evaluation of Media: Individuals need to be educated to critically evaluate online content, especially if it seems sensational or too good to be true. Look for inconsistencies in media, including unnatural features visible artifacting and what some call “uncanny valley”.
• Source Verification: Always try to verify the source of information. Is it from a reputable news outlet or official channel? Be wary of content shared widely on social media without clear attribution.
• Awareness of Impersonation Tactics: Understand that AI can be used to impersonate executives, colleagues, or public figures. For sensitive requests, especially those involving financial transactions or confidential information, use out-of-band verification (e.g. a phone call to a known number, or an in-person check if possible) before acting.

3. Regulation and Policy:

• Developing legal frameworks and regulations to address the malicious use of deepfakes, including issues of consent, fraud, and defamation.

4. Public-Private Cooperation:

• Encouraging collaboration between government agencies, research institutions, and private sector companies (including social media platforms and tech developers) to share information, develop standards, and implement safeguards.

Individual Precautions:

The DHS highlights different phases and threat actors, however, the core advice for individuals to protect themselves includes:

• Be Skeptical: Approach unsolicited communications or unusual requests with caution, even if they appear to come from a known person.
• Verify Identity: If you receive a suspicious video call or audio message, try to verify the person’s identity through a different communication channel that you know is legitimate. Ask questions that only the real person would know.
• Look for Tell-Tale Signs: While deepfakes are getting better, some artifacts may still be present:
  • Unnatural eye movements or lack of blinking.
  • Awkward facial expressions or lip-syncing.
  • Blurring or distortion, especially where the face meets the hair or neck.
  • Strange lighting or skin tones.
  • Audio that sounds robotic, has an unusual cadence, or lacks emotional depth.
• Report Suspected Deepfakes: If you encounter a malicious deepfake, report it to the platform where you saw it and, if appropriate, to law enforcement.

AI-driven impersonation is a rapidly evolving threat that poses significant risks across personal, commercial, and societal domains. As the technology becomes more accessible and sophisticated, the potential for misuse grows. By understanding how these deepfakes are created, learning from real-world incidents, and adopting robust mitigation strategies that combine technological solutions with critical human awareness, we can better defend ourselves against this new wave of digital deception. Staying informed and vigilant is our first and most crucial line of defence.

The post The alarming rise of AI impersonation: When seeing and hearing isn’t believing appeared first on ProtectionWeb.

Who are we? MWR CyberSec is a South African cyber security consultancy with research and bespoke consultancy at the core of what we do. Solving our clients’ unique cyber security challenges and ensuring they are more resilient to cyber-attacks is why we exist and continue to exist.

Some foundational concepts are useful for fully understanding cyber-attacks, so we’ll speed run them. The Lockheed Martin Cyber Kill Chain is a model for identification and prevention of cyber intrusions activity. The model identifies what the adversaries must complete in order to achieve their objective and, at a high-level, details the seven steps that Advanced Persistent Threat (APT) groups often follow during their attacks.

An APT group is a highly skilled and organised team of cyber attackers that conduct prolonged and targeted campaigns to infiltrate specific organisations or nations, aiming to achieve various goals such as: data theft, financial gain, disruption of operations, intelligence gathering, espionage, amongst others.

For APTs, so-called zero-day exploits are the holy-grail because they enable these attackers to infiltrate systems undetected, maintain prolonged access, and extract valuable information without triggering security defences and in a way that bypasses protection mechanisms wholesale. A zero-day exploit is a vulnerability that is unknown to developers and has no available fix, making it a prime target for cyber attackers to exploit before it is patched. As a side-note, to quantify their value, depending on the software systems targeted, zero-day exploits can easily sell for $10 million on exploit development markets.

Cyber-attacks to further a nation states goals have for a long while been the ideas of movies and fiction. However, in the near past (the last decade) this has rapidly moved from an ephemeral idea into real-world actions and scenarios that could play out in pursuit of a nation’s geopolitical goals. We have taken some time in our introductory article to unpack some of these below.

Around 2010, one of the most prevalent examples of cyber warfare and the weaponisation was in full effect, namely the Stuxnet worm. Stuxnet was a powerful computer worm, designed by U.S. and Israeli intelligence, that was used to derail a key part of the Iranian nuclear program by destroying the centrifuges that Iran was using to enrich their uranium. When Stuxnet infected a computer, it would check if it was connected to specific types of Programmable Logic Controllers (PLCs) manufactured by Siemens. PLCs are a fundamental element of many industrial control systems, in this case uranium centrifuges. If no PLCs were detected, the worm did nothing. However, if PLCs were detected, Stuxnet then manipulated the PLCs, which would result in the centrifuges being spun irregularly, and thus damaging or destroying them. After deploying Stuxnet, it was successful and ultimately set the Iranian nuclear program back approximately 2 years.

The only reason Stuxnet was discovered, was because it accidentally spread beyond the Iranian nuclear facility. One of the more notable bits related to Stuxnet is that PLCs are commonly air gapped (i.e. disconnected from external networks, especially the internet) as a hard defensive mechanism. Stuxnet was coded to spread via USB and it would spread to the computers controlling the PLCs via this mechanism. The fundamental take-away from this attack is that even highly secured areas, against an adversary that is dedicated and committed to the cause, are hard to detect and defend successfully 100% of the time.

In a similar manner, could a relatively innocuous piece of tech be used in ways unexpected. A simple question jumps to mind, is it possible to hack a Jeep Cherokee wirelessly from a number of kilometres away? The answer is yes. It was done by two researchers in 2015 who put a journalist behind the wheel and demonstrated their control over the vehicle from a distance. The journalist recounted the experience vividly: “I was driving 70 mph on the edge of downtown St. Louis when the exploit began to take hold”. The researchers were able to fully take control of the vehicle using only a laptop and an internet connection, leading to them toying with the likes of the air-conditioning, radio, and windshield wipers. But it goes beyond this, the researchers were able to cut all power to the drive train rendering the car useless on the highway and no means to move. They were still able to take this a step further by cutting the Jeep’s brakes, leaving the journalist frantically pumping the pedal as the SUV slid uncontrollably into a ditch.

Furthermore, the control of the vehicle enables surveillance too, providing the capability of tracking a targeted Jeep’s GPS coordinates, measuring its speed, as well as dropping pins on a map to trace its route. If you want to see more, the full video of this endeavour was uploaded to WIRED’s blog. This is an interesting attack avenue as it explores hacking cars, which just about everyone uses in some shape or form on a daily basis and are incorporating more of this sort of technology into them, but regardless back to the more military focused incidents.

A huge step-up in the cyber warfare area has been playing out as part of the Russia Ukraine conflict. The best example of this is the suspected Russian attack on the Viasat KA-SAT network that shut down communications that Kyiv, Ukraine heavily relied upon. The Russian operation resulted in an immediate and significant loss of communication in the earliest days of the war for the Ukrainian military, which relied on Viasat’s services for command and control of the country’s armed forces. The interesting part, this attack was initially launched 1 hour before Russians invaded Ukraine in February 2022. The attack was executed with a new strain of wiper malware called “AcidRain” that was designed to remotely erase vulnerable modems and routers. In the cross-fire of this targeted attack, remote control of 5800 wind turbines belonging to Enercon in Central Europe was also affected. This scenario demonstrated a real example of how cyber-attacks can be targeted and timed to amplify military forces on the ground by disrupting and even destroying the technology used by enemy forces. The Danish defence minister stated that, “The cyber threat is constant and evolving.

Cyber-attacks can do great damage to our critical infrastructure, with fatal consequences”, further highlighting the importance of cyber defence and incident response capabilities. These capabilities don’t explicitly end at the terrestrial. There are companies who can be incredibly technically advanced in the solutions they provide; however, they are not immune to cyber breaches. Cyber security is often neglected until the inevitable happens, they are breached. Russia has attempted attacks against the Starlink systems in order to jam the internet service in Ukraine.

A Belgian cyber security researcher was able to breach Elon Musk’s Starlink satellite system using a simple Raspberry Pi device in conjunction to other electronic components costing the equivalent of 500 ZAR. This is a prime example of the fact that hacking does not always follow the “traditional” path of compromising an individual’s computer or an organisations server, but can instead begin with targeting embedded electronic systems in hardware-based attacks.

A notable ATP group to mention is that of the “Volt Typhoon” group, which has been linked to the Chinese government and active since 2021. The existence of this particular APT group surfaced publicly in May 2023, when Microsoft reported that the group, had targeted US critical infrastructure in espionage operations, and lay dormant within their infrastructure for as long as 5 years. Specifically, this APT group was targeting Operation Technology (OT) systems using zero-day exploits, to pre-position themselves for future attacks of sabotage. This group even demonstrating critical capabilities in compromising Microsoft to see the level of detail that the organisation had on them, during the course of Microsoft raising these events publicly.

In early 2024 the Cybersecurity and Infrastructure Security Agency (CISA) released an advisory that raised concerns about the potential for these threat actors to use their network access for highly disruptive effects in the event of potential geopolitical tensions and/or military conflicts. It is critical to note that no impact from this attack was witnessed in any of the systems they were found to have compromised. This largely indicates a co-ordinated effort, from a highly sophisticated state sponsored actor pre-positioning for maximum damage to enhance geopolitical outcomes. It was noted through the investigation that activity related to this threat actor had gone back, in some cases, in excess of 10 years. Once again highlighting why cyber defensive capabilities are so important. Can you imagine the fallout had this ATP group been able to disrupt critical infrastructure, power stations, water supplies and government infrastructure, within the United States.

MWR has long spoken of the need for organisations to assume that at some point they will be breached, and prepare as such. The reality is an organisation can only prepare for compromise, by having a robust, encompassing and sound cyber security strategy and a capable suite of armaments to deal with this unfortunate scenario. The three pillars of people, process and technology need to be soundly practiced, complementary in all they do and have the ability to rapidly work to contain, eradicate and recover from a cyber security compromise. Unfortunately, in our experience, often the hardest phase for an attacker to complete is the initial phase to gain access. Once in a target environment it is far too common that organisations have minimal controls and restrictions within their internal network. We therefore advise our clients to work backward. Understand what you have that an attacker would want access to, and structure controls, defences and barriers from this point outward towards the external perimeter. Performing this exercise will give you a view you never had of your internal network and how an attacker is likely to target you. This ultimately makes you more robust against their attacks.

From MWR’s experience, what we can say is that you do not want to end up in a position where you have to perform incident response activities to eradicate the threat actor. However, you want to prepare as if you will end up there someday. It can prove to be a very challenging endeavour whilst potentially incurring financial and reputational damage as a result of such a breach. As an example, IBM’s 2024 Cost of a Data Breach report shows data breaches in South Africa cost 53.10 million ZAR per incident, on average. The main driving factors of these costs are business disruption, post-breach customer support as well as remediation. In addition to this non-compliance with regulations also contributes to this number. It is substantially easier to be proactive when it comes to security rather than being forced to become reactive to such events.

Companies developing new technology face near constant attempts to breach their security. Unfortunately, those companies without mature cyber security environments may never realise a breach has occurred, let alone actually respond in an appropriate manner until it is too late. Real-world attackers essentially have “unlimited” time, and if advanced and persistent enough, they will find a way in. Would you be able to respond and eliminate the threat before substantial amounts of damage can be caused, or would you rather take the proactive approach and diminish the chances of ever having to be in such a position?

The post Tactical Typing: The Rise of Cyber Security in Modern Warfare appeared first on ProtectionWeb.