I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up. Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991.... Weiterlesen
Intelligence View
AI crawler user agents are self-reported: 468 real fetches, 991 fake ones
I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up. Then I re-cut eight days of logs by…
SOCIAL SHARE CARD GENERATOR