Gaming the system

The internet will inevitably steal the soul of every child at some point as he or she grows up, but some are speeding the process along by manipulating YouTube's powerful algorithm.

Creepy, weird, and violent clips intended for kids pop up frequently on YouTube and even YouTube Kids, its family-friendly app designed for children that launched in 2015. And YouTube’s policy is to be mostly hands off — until someone complains.

Take for example the video embedded below that showed up on both YouTube and the Kids app: You press play and at first your screen fills with recognizable cartoon characters and cheesy music — but things take a drastic turn when Elsa and Spider-Man arm themselves with automatic weapons.

Alongside kid clones of themselves, the Hulk, Batman, Elsa, and Spider-Man kill baddies like the Joker for stealing guns from their tiny twins.

For nearly 30 minutes more the disturbing video oscillates between violent episodes like this and singalongs to "If you're happy and you know it."

Upload whatever clip you want with the right keywords, and your inappropriate content could land in front of kids' faces. This seems to work especially well if you have other kid-friendly content on your page.

The Elsa shootout above is incredibly bizarre, even for YouTube. It mimics tropes and trends on YouTube Kids in order to lure young viewers into watching. The video's title is "Bad Kids & Learn Colors Giant Watermelon Finger Family Song Nursery Rhymes | Learning Video For Kids." That may sound like a bunch of nonsense, but if you have a kid who watches YouTube, it's algorithm gold. Bad Kids, Learn Colors, Finger Family, Nursery Rhymes, and Learning Video For Kids are all popular search terms, which YouTube and its Kids app generally reward with prevalent placement.

With over 797,000 subscribers, the channel Battle VS Death Battle(opens in a new tab) hosts the video along with dozens of others like it. The Elsa shootout has over 5 million views, and was uploaded last month. Considering it was accessed on the Kids app, it was available for viewing by millions of children. The app has been downloaded 50 million times worldwide from the App Store and Google Play since its release, according to app analytics from marketing firm Sensor Tower.

The Elsa shootout video, along with other questionable content, was removed from the Kids app after we contacted YouTube for comment but remains on YouTube. Videos can live on both the app and YouTube itself.

Since its inception in 2015, YouTube has repeatedly marketed the safety(opens in a new tab) of its Kids app. But YouTube and its Kids app rely on user reporting, so unless someone flags a clip, it could potentially be seen by millions of kids. YouTube knows it has an issue with weird videos slipping through the cracks; when you first install the app you're greeted with this reassuring message:

The content selected to be featured on the YouTube Kids app is chosen by algorithmic filtering, user input, and human review — YouTube has tried to spin this as a positive(opens in a new tab) combination, but it does let inappropriate videos slip through the cracks via search. Videos that come up in search results in the Kids app are selected by an algorithm and are not subject to human review, according to Google's support page(opens in a new tab) for YouTube.

"We work to make the videos in YouTube Kids as family-friendly as possible and take feedback very seriously. We appreciate people drawing problematic content to our attention, and make it easy for anyone to flag a video," a YouTube spokesperson said in a statement. "Flagged videos are manually reviewed 24/7 and any videos that don't belong in the app are removed within hours. For parents who want a more restricted experience, we recommend that they turn off the Search feature in the app." 

Considering kids videos are among the most-viewed clips on YouTube, there is big money to be made. According to the third-party ratings tracker Social Blade(opens in a new tab), the Battle VS Death Battle channel may bring in anywhere from $92,000 to $1.5 million annually from ads(opens in a new tab). With disabled comments, no associated social media accounts, and no contact information, the channel is clearly trying to fly under the radar.

Given the kid-focused search terms used in the Elsa shootout video's title, it's unlikely the creepy video and others like it were made for adults with some very odd cartoon interests. Plus, the channel is managed by the network Mango Kids, which describes itself as an"easy learning destination for your young ones." Most of the other channels managed by Mango Kids do seem innocent enough, although totally weird.

Mango Kids, which is based in India, didn't respond to our request for information on its operations, channels, and target audience. But it's not alone, there are several YouTube channels that create questionable content for kids using similar style, characters, songs, or SEO keywords. (Just check out the video below.)

While some parents may be O.K. with cartoon violence, others have been voicing their concerns about YouTube videos for a while now(opens in a new tab).

"Yeah, I've totally stopped letting my toddler watch YT or YT Kids because there are too many creepy videos," one parent wrote on Reddit(opens in a new tab), in reference to the channel Toy Freaks(opens in a new tab), a bizarre channel that boasts 8.1 million subscribers.

"I download the YouTube videos I've prescreened for mine to watch, unless I can sit and watch with her," another wrote.

A quick search for the word "gun" on the Kids app showed this tutorial(opens in a new tab) on how to build a real coil gun. The clip is clearly not intended for children, but somehow it was filtered into the Kids app, which is yet another example of how algorithms can cause trouble. (Other inappropriate words, such as "sex" and "naked," were filtered out on the app.)

While parents can easily report a video and get it removed from the app, it has an overwhelming amount of content, and removal could conceivably happen only after your child has watched the offending video. Turning off the search function limits the videos your child sees, and that does appear to help — but it severely restricts the number of videos in the app.

If you can't monitor every single thing your child watches, a much safer option is to use a curated service. YouTube's user-generated content doesn't go through a vigorous review process like the kids section on Hulu, which does.

But even a traditional content review process won't always catch everything. Netflix faced backlash last month when a parent noticed a NSFW drawing in one of its cartoons and tweeted about it, which led to it being pulled(opens in a new tab). 

Huge tech companies like Facebook, Twitter, and YouTube have all faced criticism for putting too much faith in their algorithms and letting the community monitor itself. Their reach proves time and again to be too large for an algorithm alone to police and, unfortunately, people will continue to be terrible on the internet. Even if these companies were to monitor content more closely (as Facebook has promised to do(opens in a new tab)), the community may push back.

Thanks to the internet being in everyone's pockets, there is never going to be a filtration device or a setting to completely block your kids from seeing something inappropriate for their age. The internet can be both a positive tool, and a cesspool. That's something every parent should discuss with their kid, when the time is right.

While options are limited in the long run, at least parents have some power when it comes to shielding kids from creepy YouTube videos: If turning off the search function doesn't go far enough, delete the Kids app and block YouTube's site(opens in a new tab) from your kids' devices.

User-generated content comes with a risk, but parents don't have to take it.