Content Connector Banned

Marfeel maintains an intricate and intelligent content extraction strategy. It's engineered to efficiently crawl and extract the relevant content a publisher produces with accuracy so it can be delivered to their audience as quickly as possible.

Because of Marfeel's commitment to its rigorous extraction policy, Marfeel crawlers can at times be mistakenly banned by publishers' servers.

 

Main Reason

The main reason Marfeel's crawler can be banned is because Marfeel repeatedly queries publishers' servers to accurately update the content on their Marfeel mobile site. 

When publishers' servers see repeated and identical requests from the same user-agent - in this case, Marfeel's crawler - they can confuse it with an attack or hacking attempt such as a DoS attack, and ban the server. 

Banning Marfeel's crawler could also be due to a customer's use enhanced security measures like an anti-intrusion system or firewall.

Solution

Marfeel's crawler is identified by the internal Marfeel device and operating system it's run by. When Marfeel's crawler is banned, the User-agent is changed to MarfeelMan.

The publisher must then whitelist the new User-agent "Marfeel-crawler" to grant access to its server once again. This prevents the crawler from being banned in the future, and allows Marfeel to crawl for and extract content. 

For a complete overview of how Marfeel extracts content, see the Content Connectors article. 

For more information regarding Marfeel's effect on customers' servers, see the Impact on a Publisher's Server article.