Here's what the AI tool (Copilot) says about it. Kind of self-conflicted since it may be one of the bot-masters doing the scraping. This is all beyond me, but it does use some good buzz-words! Half of this is probably wrong, but it sounds good to my ignorant self.
Every time the site goes down, I worry about losing the massive collection of searchable Pontiac information contributed by real people with real experience. I guess if these AI monsters are collecting all the information, it somehow won't get lost? But I suspect it will get distorted by combining with a lot of nonsense.
----------------------------------------------------------------------------
"🧠 What’s Likely Happening
• Web scrapers and crawlers: These bots scan forums for content, often ignoring polite crawl rules like robots.txt. Some are search engine bots (Google, Bing), but others are aggressive scrapers used by AI models or data harvesters.
• Spam bots: They create accounts or post junk content. Even if they don’t succeed, their repeated attempts can overload login and posting endpoints.
• Click bots and ad fraud bots: These simulate traffic to manipulate ad impressions or SEO rankings.
• AI training bots: As discussed in this thread on the forum, many users suspect that AI tools are scraping discussions to train models—especially technical threads like camshaft recommendations.
🔍 How to Gain Insight
Here are a few ways the forum owner or admin could investigate:
• Server Logs: Analyze Apache or Nginx logs for:
○ IP addresses with high request rates
○ User agents that mimic browsers but behave abnormally
○ Requests to endpoints like /search.php, /login.php, or /newthread.php
• Bot Detection Tools:
○ Services like Cloudflare Bot Management or Imperva can classify traffic as human, good bot, or bad bot.
○ They offer dashboards showing bot types, IPs, and behavior patterns.
• Behavioral Analysis:
○ Look for traffic with high bounce rates, short session durations, or repetitive patterns.
○ GA4 or server-side analytics can help identify anomalies.
• Honeypot Fields:
○ Add invisible form fields that bots tend to fill out. If triggered, flag the session as bot traffic.
• Rate Limiting and CAPTCHA:
○ Throttle requests from suspicious IPs or require CAPTCHA for guest access during traffic spikes.
🧩 Why It’s Hard to Pinpoint
Bots often rotate IPs, spoof user agents, and mimic human behavior. Without active monitoring or bot mitigation tools, it’s tough to know which ones are causing the slowdown. But the forum’s own users have noticed patterns—like 1000+ “guests” viewing just a few threads—which strongly suggests automated traffic."
|