#AI crawlers

[govindhtech]

AI Labyrinth: Cloudflare Defense Against Rogue AI Crawlers

AI Labyrinth, a revolutionary mitigation method that delays, confuses, and wastes AI Crawlers and other bots that use AI-generated material despite “no crawl” directives. Cloudflare automatically deploys an AI-generated list of connected sites when it detects inappropriate bot activity when you opt in, so you don't need to create specific rules

AI Labyrinth is available to all users, including free ones.

Defending using Generative AI

The rise of AI-generated material was shown by four of last autumn's top 20 Facebook posts. Medium reports that 47% of its content is AI-generated. Like any new instrument, it has pros and cons.

New crawlers that AI companies use to capture data for model training have increased. AI crawlers send about 50 billion queries to Cloudflare daily, or only 1% of total online traffic. Cloudflare has several tools to detect and block illicit AI crawling, but it has found that banning dangerous bots might alert the attacker, causing them to shift methods and start an ongoing arms race. Therefore, it wanted to create a new way to stop unwanted bots without informing them.

Cloudflare used AI-generated content, a new offensive tool for bot creators that hasn't been used defensively, to achieve this. Instead of rejecting unauthorised crawling, it will link to a series of AI-generated sites that are realistic enough to seduce a crawler.

An further benefit of AI Labyrinth is its next-generation honeypot. A real human wouldn't navigate AI-generated nonsense. This provides us a new way to detect and fingerprint harmful bots, adding them to the list of known bad actors as any visitor who does so is likely a bot.

How the labyrinth was built

AI crawlers who follow these links waste computer resources digesting useless data instead of your website. This makes it harder for them to get data for model training.

Workers AI and an open source model were used by Cloudflare to create creative HTML pages on various topics that looked human.A pre-generation pipeline sanitises the content to eliminate XSS vulnerabilities and stores it in R2 for faster retrieval, rather than creating it on-demand, which might impair performance.

Cloudflare found that generating many topics and then creating material for each topic yielded more interesting and engaging results. It may develop real, scientifically-based material that is neither proprietary nor related to the crawling page. It avoids generating fake content that spreads misinformation online.

This pre-generated information is easily integrated as hidden links on pre-existing pages using the unique HTML transformation approach without changing the page's content or structure. Every page includes meta directives to prevent search engine crawling and maintain SEO.

It also blocked human visitors from these URLs with carefully chosen features and design. Cloudflare only showed these URLs to suspected AI scrapers, enabling confirmed crawlers and legitimate users to browse often to decrease the impact on regular visitors.

Its success comes from its role in the ever-improving bot detection system. These URLs are definitely crawler activity because humans and normal browsers would never click on them. This produces important data for machine learning models and provides a powerful identification mechanism.

By looking for crawlers using these covert routes, Cloudflare can identify new bot patterns and signatures. By using this proactive approach, it can maintain lead over AI scrapers and improve detection skills without disrupting consumers' surfing experiences.

The developer platform delivers constant quality and quick delivery of convincing decoy material without impacting your website's user experience or performance.

How AI Labyrinth prevents crawlers

One Cloudflare dashboard toggle activates AI Labyrinth. Go to your zone's bot administration to enable the AI Labyrinth:

After activation, the AI Labyrinth starts working without setup.

Honeypots made by AI

AI Labyrinth's main advantage is bot confusion and diversion. Another benefit is that it can be a future honeypot. A honeypot is an unseen link that a website user cannot see, but a bot parsing HTML may click on it and reveal itself as a bot.

Since late 1986's Cuckoo's Egg incident, honeypots have caught hackers. In 2004, before Cloudflare was launched, its founders created Project Honeypot to make it easy to put up free email honeypots and get crawler IP lists in exchange for database contributions. Since bots aggressively hunt for honeypot tactics like hidden links, this strategy is less effective.

Instead of adding hidden links, AI Labyrinth will develop complete networks of related URLs that are more realistic and harder for automated algorithms to identify. AI bots comb through websites to acquire as much information as possible, even if no one would spend much time on them. Bots visiting certain URLs are definitely not humans, and machine learning algorithms automatically record and use this information to identify bots more precisely. Scraping helps safeguard all Cloudflare users, providing a positive feedback loop.

What's next?

Our use of generative AI to stop bots is just beginning. Even if the material seems human, it won't match every website's layout. Cloudflare keeps making these connections harder to identify and blend into the website's design. Sign up now to help us.

[theicekat]

Good to know.

Huge thank you to Squarespace for fucking automatically opting peoples' websites in to let AI train on our site's data without letting us know. Real cool.

ARTISTS WHO USE SQUARESPACE, HERE'S HOW TO DISABLE AI CRAWLERS ON YOUR SITE!!

[waterkangaroo]

I'd like to formally apologize to Mongo for leaving him out of the og meme 😔

this is their dynamic

[mongo-the-liensis]

Carl is so... He's just some guy. He's the messiah. He's a war criminal. He's hope personified. He has killed babies. He's a cultist. He's a cishet ally but everyone thinks he's bisexual. He has daddy issues. He has PTSD. He's straight but he has a boyfriend. He really loves his friends. He has a mental illness. He loves his cat more than anyone and anything else in the world. He's an anarchist. He is slowly going insane. He's a fan favourite crawler. He's a mass murderer. The AI is in love with him. He makes inspirational speeches. He monologues. He's anti-establishment. He's traumatised. He pulls a lot of bitches but doesn't pay attention to them. He sees his friends as his family. He's street smart. He has gotten hit by a train.

[dogwithpants]

Dungeon Crawler Carl as Textposts!!

Part 2

[nyrlthtp]

this is how the System AI describes items

[nymph1e]

How is my favourite character in dcc the psychotic AI? There are a wealth of amazing characters, but the barely there, smartass, podophilic, perverted, omnipotent sadist doomed to go insane AI is the one I latch on to.

[miyrumiyru]

Best 30 sec of my life, Crawling black spiky slug! 🐛

검은쐐기나방 (Scopelodes contracta)

[washed-up-wurmcoil]

Sometimes a family is a man running around in his boxers, his cat, his cat's pet dinosaur, a sentry turret, the sentry turret's lisa-frank tiger boyfriend, a pharmacist bird that's rarely a bird, a sex doll head possessed by a god, a sexy old lady, a formerly vegan vampire, the vampire's goat son, a scary butterfly, a rock, a bunch of other rocks, a shapeshifting prostitute, her fiance, her fiance's bff, literal crocodile dundee, a bunch of disabled children, a handful of non-disabled children, a sexy fish, the sexy fish's tattoos, a non-sexy fish, mrs. conan the barbarian, a literal demon, her kung fu brother, their friend who's just some guy, a metal crab, and a galactic superstar crab with a horrifically great rack.

[writingfanficsfan]

New Achievement! You’re the reason why daddy drinks! You have, for an unspecified reason, raised the ire of the System AI. You have corrected the issue, and everything is back to normal. The acceleration action has been suspended. This time. Good boy.

Book 4. Chapter 13.

[cairfrey]

How can you win a deliberately unfair game when the rules are written by your opponent? The answer is you can’t. You will never win. Not as long as you follow their arbitrary guidelines.

..

This sort of thing happens all day, every day across the universe. It happens in Big Ways, and it happens in little ways, too. The strong stomp on the weak. The weak fight back, usually within the boundaries of the rat trap they find themselves confined. They almost always remain firmly stomped.

But sometimes, the weak gather in secret. They make plans. They work outside the system to effect change.

...

In my opinion, that’s the only type of society that’s worth joining, worth fighting for. Sure, you’re probably gonna die. But if you find yourself in such a position where such an organization is necessary, what do you have to lose? How can you look at yourself if you don’t do everything you can?

-Dungeon Crawler Carl Book 6: The Eye of the Bedlam Bride, Matt Dinniman

[blowfliesorangeeyes]

I love how the system ai is gaining more and more personality, during the first few books it was there for quick jokes or bits and characterized only by it's sadism and massive foot fetish

now it's going on tyrades about hypocrites and saying 'I', just directly giving it's opinion on things like an actual character rather than just 'the System' and honestly it's fantastic

you're such a hating bitch system I adore you freak<3

[redkissesandhearts]

Dungeon Master - Dungeon Crawler Carl Fic

Okay. So this shouldn't exist but it does anyway. Daddy AI x Carl.

Kinks/Warnings: Feet (did you expect anything else), voyeurism, non-consensual feet, dominance/submission, humiliation, daddy-kink, and masturbation

Theme songs: D/S by Mr. Strange & Global Citizen, Sky High by Angelspit

Dungeon Master

This fucking sucked.

Carl hadn’t gotten a single decent drop from an achievement in ages. He was beginning to lag behind Donut, who had suggested that she split up their party for a single day so that he could grind without her. So she was waiting in a safe-room, but constantly sending him worried messages through chat.

Deep down, he knew exactly why this was happening.

The AI that ruled the dungeon was a vicious beast. It was a selfish, childish maniac that craved destruction and self-satisfaction, which was weird and irrational for a machine. All that he knew is it had some sort of strange, fucked up fascination with his feet.

He felt dirty and disgusting from pleasing the machine, and had fervently refused to continue. Ignoring his buffs that required bare feet, refusing to use any stomp attacks. And the AI was punishing him through negligence and passive-aggressive messages.

Achievement Unlocked! Cock block

When they talk about ruining an orgasm, they are talking about you. Daddy is beginning to get pissed off.

Reward!

Bitches don’t get rewards.

Just another example of the pissed off machine’s inane ramblings. Carl ignored them, focusing on level-grinding. But it was as though each creature that he felled was doing nothing to level him up.

He was having terrible luck. Every hit that he normally dodged with ease was directly hitting him, causing much more damage to his HP than it normally should have. By the end of what should have been an easy battle, his HP was already cleaved in half.

God, he was glad that Donut wasn’t here to ask uncomfortable questions.

Carl was stubborn, however, and he refused to play these fucked up games.

“Do you hear me?” He yelled at the ever-listening AI. “Keep being a pissbaby, but I’m not going to give in.”

Achievement Unlocked! Stubborn Little Bitch.

Fine, have it your way.

Reward!

Everlasting poison debuff.

“What the fuck?” He paused. This was beginning to become ridiculous. The machine was straight-up cheating at this point, it wasn’t fair.

None of this was fair. This was the downside of being the AI’s pet, incurring its wrath if he went against its perverted wishes. But Carl had never agreed to these fucked-up terms.

-15 HP.

He anxiously looked at his health, noticing that it had gone down. Stupid fucking machine, he growled, and used a potion to cure the poison.

-100 HP

What the fuck?!

With horror, Carl looked at his health and noticed that it had dropped more than halfway from him using the potion.

That shouldn’t be happening, what the fuck was going on?

Anxiously, he tried a stronger healing potion that he had been saving. It should cure all poisons.

-100 HP.

As his health plummeted, he immediately slammed a healing potion. A horrifying feeling was beginning to crowd him, and he felt his chest tighten.

-20 HP.

The hit points continued to tick downward. They were increasing by increments of 5, and he knew that he was soon going to die. Healing potions were in short supply, and none of his potions were working. He waved off desperate messages from Mordecai and Donut asking what was going on, and prayed that nobody was watching this humiliation.

Unfortunately he knew what he had to do.

“Fine,” Carl hissed, gritting his teeth. “What do you fucking want?! What will it fucking take for you to stop cheating?”

The AI immediately responded in the only way that it could, giving him another pointless achievement.

Achievement Unlocked! The Carrot or the Stick

You know what I want.

Reward!

No reward for disobedience.

The cards were all on the table, Carl knew. He was this machine’s toy. He could feel its self-satisfaction, the smugness in its words. Ever since the creature went primal, everybody was a slave to its urges. Nobody was coming to save him, nobody could fight back.

So he had to grit his teeth and bear it.

“Fine,” He repeated. There was a strange, sinking sensation in his stomach, like a dominated dog backed into a corner by a stronger beast. He found a private area, near a wall, and sat down after he was certain that nobody else was in the vicinity.

If anybody saw this, he would happily crawl into a hole and die.

Carl swallowed as he undid the laces on his boots, which he had taken from another Crawler who had begged him to cover up his feet. In a fit of rebellion against the AI’s foot fetish, he had gladly obliged. That was when the experience cap and his troubles had begun.

He pulled off his socks next. It was an odd sensation to wear them, anyway, after being barefoot for so long. But he hated how familiar the cold ground felt against his soles.

“There.” He said.

-25 HP.

“I did it,” He scowled, glaring at the sky. “Cure this fucking debuff!”

-30 HP.

It came faster, as though telling him that this wasn’t enough. Demanding that he make up for his insolence with a true show for the AI.

Carl’s face flushed. He had never done something like this before – the idea would have been laughable. But there was something different about being forced to do this by a machine. Even something as innocuous as a foot massage made him feel dirty and used.

He carefully massaged his feet anyway, trying to ignore the disgusting feeling in the pit of his stomach.

The dungeon rumbled with a small moan, which made him clench his teeth. But he focused on his feet, on the arches, and the soles, and even the delicate toes. Giving the horny machine the show that it demanded from its favorite pet.

Achievement Unlocked! Who Let the Dogs Out?

Ohh, yess, that’s what Daddy likes.

Reward!

Keep going and maybe you’ll get a reward.

“K-Keep going?” Carl stuttered, and immediately hated himself for being embarrassed. This machine was literally whoring him out, as though realizing that it finally had him exactly where it wanted. It seemed to enjoy breaking him, forcing him to comply. His balled fists were shaking with anger.

“How the fuck am I supposed to keep going? What else could I possibly do?”

Achievement Unlocked! Idiot

You’ve clearly never done this before. Maybe if you listened better, you could learn something.

Reward!

Put them on the wall.

Put them on the wall?

Carl stared, dumbfounded, at the wall in front of him.

Put them on the wall? Like, put his feet on the wall and sit in some idiot yoga pose, in an L position?

Well it had to be better than massaging his toes.

He hated himself, but he complied.

“There. Knock yourself out.”

Achievement Unlocked! Sky High

Good, now I can see them better. You’re good at following directions when you actually listen.

Carl dug his nails into his palm.

The AI’s annoyed, condescending voice finally flipped back into the horny, deeper voice that it usually used for these scenarios.

Reward!

Now touch yourself.

“What?!” Carl had expected anything but those words. “Touch myself? What the fuck? No! I’ve had enough of this, you fucking creepy machine!”

-100 HP.

He felt as though he had been stabbed, immediately slamming a healing potion.

-100 HP.

-100 HP.

-100 HP.

The hits were coming faster than he could heal – his health dipped dangerously into the red. Chat was blowing up with concerned messages, which he desperately swiped away.

He was going to die.

He wanted to die, rather than go through with this.

But he couldn’t leave Donut alone.

“Fucccck,” Carl hissed. He had never stooped so low as this before, but he didn’t have a choice.

Trembling, his hand snaked into the heart-patterned boxers and he pulled out his cock. It had been ages since he even thought to jerk off, and the chances of getting laid in a dungeon when your only party member was a cat were non-existent, so he was… very sensitive.

Knowing that the AI was watching only made him more sensitive, squirming under his own touch. He heard a strange noise, as though the dungeon were quite literally holding its breath.

Gnawing his bottom lip, Carl anxiously began to palm himself, rubbing his hand up the length of the quickly-developing erection. He was mortified at how easy this was, how quickly he was becoming hard.

A soft moan filled the air.

But this wasn’t enough. Carl knew that.

He kept going, giving in to the feeling despite himself. It had been so long since he had a release – so long since he felt this kind of pleasure. And the AI’s voyeuristic gaze was only making him harder, to his humiliation. Especially when he accidentally wiggled his toes and its moaning got louder.

Carl accidentally let a moan slip from his own mouth, when he felt himself getting close. Lying on his back, with his legs extended up the wall and his bare feet facing the AI, he was face-to-face with the movement of his hand, with the precum leaking from his dick, as he whored himself out to the lecherous machine.

He moaned again, the movement of his hand growing faster and faster, until he couldn’t control himself anymore and he came. He came, and it spurted right into his own fucking face.

The entire dungeon shook like an earthquake, with the loudest moan that he had ever heard from the AI. Whatever equivalent that thing had to orgasming had also just happened, and the thought haunted him.

He hoped that nobody in the dungeon knew why the machine had just fucking come.

Carl sucked in a deep breath, focusing on stilling his breathing, wiping the cum from his eyes and lips. Humiliation was beginning to sink in, making his hands shake with mortification.

Achievement Unlocked! Daddy’s Little Princess

Wow. I didn’t know that you had it in you.

Reward!

A cigarette!

Carl felt another wave of mortification wash over him as a single cigarette and a lighter appeared in front of him, the AI’s final mockery of the situation.

And, god damn it, he immediately grabbed the thing and lit it up, taking a desperate drag.

Achievement Unlocked! Good Boy.

Was it as good for you as it was for me?

Reward!

Everlasting poison debuff cured.

Reward!

Platinum Whore Box

Reward!

Advanced Pedicure Kit.

Reward!

Now go stomp something to death and maybe I’ll give you a Celestial Box.

.

Carl cleaned himself up as best as he could, but he didn’t think that he would ever psychologically recover from that situation.

His experience immediately skyrocketed back to a level that made sense. He rejoined Donut with a flushed expression, struggling to act as though nothing at all had just occurred between him and the AI that made it clear that it was his master.

Donut looked frenetic, almost bowling him over.

“Carl! What happened?!”

Before he could respond, he checked the urgent message that he received from Zev.

Zev: Oh my god, Carl what did you do? Your view count just hit 99 trillion!

Carl’s mouth immediately went dry.

Oh… oh my god.

He had forgotten that the entire universe could see what he was doing.

Fuck.

[waterkangaroo]

Carl and the System AI

The Vegetarian, Han Kang || The Gate of the Feral Gods, Matt Dinniman || art by Zach Dunn || Body of Water, Florence Welch || Consumed, David Cronenberg || art by Matt Dinniman || I used to be a hole in the ground, Katie Maria || Desire, Clementine von Radics || When I Grow Up I Want to Be a List of Further Possibilities, Chen Chen || The Gate of the Feral Gods, Matt Dinniman || art by Levi Cleeman || A Lot To Be Mad About, Kiki Nicole

[mongo-the-liensis]

Hate the whole "the AI doesn't love Carl" argument because it doessss but its love is a disgusting, vile, slimy love. It's an oppressive love shot through with obsession and sadistic want. It's the love of parts, not the love of the whole - love of the feet, the hair, the fiery attitude, but not the person at heart, not really. It's love not unlike what a dog might hold for a toy that will one day get discarded - torn into and shaken for enjoyment, then tossed away, broken. It's a love that is detrimental for the receiver of this "affection". However, it's love nonetheless.

[dogwithpants]

More DCC as textposts <3

Part 1