Fight Web to Lead Spam w/ Akismet
Back in January, I posted about a way to help combat web to lead spam. That type of solution works well, but is not scalable. Also, it is a reactive approach rather than a proactive one.
I decided to try and see if I could incorporate Akismet into the web to lead process and I was successful in doing so! I created a set of scripts for you to download if you’d like to leverage Akismet with your web to lead forms.
Akismet is the best spam tool I have ever used. When I posted in January, it had captured 7,680 spam in its existence on this blog. Less than 4 months later, the spam count is up to 23,994. Needless to stay, spam is an exponentially increasing problem and will plague your Salesforce.com environment eventually. I feel that Salesforce.com needs to include a spam filter in their product for web to lead (vote for it).
Until then, leveraging Akismet could help you significantly.
About the Solution
The scripts are intended to be proof of concept for you to use and apply to your own environment. The code and downloads are now being officially hosted at Arrowpointe’s Open Source Project at Google.
To use this solution, you’ll point your web to lead forms to this script rather than to the standard Salesforce Web to Lead page. The script accepts the data from your web to lead form, passes it to Akismet to determine whether it’s spam or not and then passes the data to Salesforce.com’s web to lead page with the Akismet result appended to it.
In Salesforce, you’ll need to add checkbox field (e.g. “Akismet marked as spam”) on your Lead. If Akismet thinks it’s spam, that field will be set to TRUE. You would then need to add assignment rules or validation rules to do whatever you need. For example, you could have a lead assignment rule looking at that one checkbox field and put leads into a special queue if they are marked as spam.
The script leverages the Akismet PHP5 Class to handle the core communication with Akismet. I found this class from the Akismet Development page.
This script will only work on PHP5 and requires the cURL module to be enabled. cURL is enabled by default in most PHP installations. The PHP5 requirement is a limitation of the Akismet PHP5 Class. If you are on another platform (PHP4, Ruby, Java, etc.), I don’t see any reason why you couldn’t use these scripts and integrate a different Akismet toolkit into it. Additional toolkits are available from the Akismet Developer Page.
Cost
You’ll need a server running PHP5. If you have a server already setup, then hardware should cost you nothing.
Akismet is free for personal use, but has a small license fee for commercial use. If you will be using this in production, you should purchase an Akismet Commercial License Key. During development, I am sure you could use a personal key to make sure it works.
The scripts themselves are free and licensed under the GNU General Public License v3. I did this as a proof of concept for the betterment of Salesforce.com data quality everywhere.
Getting Started
- Add a checkbox field to your Salesforce.com Lead Object that will hold whether Akismet thinks it’s spam or not.
- Download the scripts. Three files will exist in the zip file:
- index.php: The main script that handles the incoming data, talks to Akismet and posts the data to Salesforce.com Web to Lead.
- constants.php: You will need to go into this file and edit some variable values based upon your own organizational setup. See the next step.
- Akismet.class.php: This is the Akismet PHP5 class I was talking about above.
- Edit the constants.php file:
- Enter your Wordpress API key where it says ENTER_WORDPRESS_API_KEY. Get a personal or commercial key if you don’t have one.
- Enter your company URL where it says ENTER_YOUR_COMPANY_URL.
- Enter your company’s Salesforce.com Org ID where it says ENTER_YOUR_SALESFORCE_ORG_ID. This is actually optional. Doing this allows you to remove the OID from your public web to lead forms so spammers don’t know your Org ID. Doing this will reduce the amount of spam you actually need to process.
- Generate a web to lead form in your Salesforce setup. Find the new Salesforce field you created in step 1 and copy the id value from the HTML form and put it in the constants.php file where it says ENTER_THE_W2L_CUSTOMFIELD_NAME. This step is required so that your Salesforce org is actually populated with the Akismet result.
- PHP Advanced: The $Akismet_noPass array holds the names of fields that should not be included in the content passed to Akismet. Feel free to add/remove values from this array. The values in the array are referring to the names of form fields in your web to lead HTML form. I have no idea if this helps/hurts, but it seemed like a practical thing to add into the script.
- Upload the scripts to your web server and note the fully qualified URL for that directory (e.g. http://www.example.com/web2lead)
- Update your web to lead forms to have them post to the location of the files from the previous step (e.g. http://www.example.com/web2lead/ - make sure to put the / at the end of the URL. Not sure why, but I wasn’t able to get it to work without it)
- Test your form to see if it works. The script acts as an intermediary between your form and Salesforce web to lead. The end-user experience should be the exactly the same with or without the scripts.
- Once you know it works, you should add a lead assignment rule into Salesforce as rule #1 that looks to see if this field is checked. If so, then route the lead to a “Potential Spam” queue or something of that nature. Another option is to create a validation rule that doesn’t even allow the lead into the system.
- Make sure your Auto Response rules don’t email a reply to leads marked as spam. If you allow this, then those spammers have an email address to try.
- Update some/all of your web to lead forms to post to this new page. If desired, remove the “oid” field from the HTML form for each of these since the script will pass your Org ID to Salesforce automatically.
Other Stuff
These scripts are a proof of concept. I am not officially supporting them, but am happy to help people out informally. Post comments here if you have questions/comments/criticisms.
I have only tested this with leads that were either real or very obviously spam. It worked well. From Akismet’s perspective, the data it checks looks just like a blog comment and I can attest that Akismet is amazing at identifying blog comment spam. So it should work well for Salesforce.com web to lead.
I am going to update my existing web to lead forms on this site and see how it works and report back to you. I encourage you to do the same and let me know your experiences with it or recommendations on how to improve the scripts.
Enjoy!
(posts)
Scott Hemmeter Said,
May 29, 2007 @ 8:25 am
I am in the process of implementing this on my own websites and found out that you must put a slash on the end of the URL in your < form > tag. For example, if your web to lead spam check script is at “www.example.com/webtolead”, you’ll need to put “www.example.com/webtolead/” (note the slash on the end) for your form to post to.
gokubi.com » Blog Archive » Akismet blocking web-to-lead spam Said,
June 1, 2007 @ 8:21 pm
[...] Hemmeter shows again just how smart he is. I decided to try and see if I could incorporate Akismet into the web to lead process and I was [...]
bigbrownboy Said,
June 18, 2007 @ 10:33 pm
I’m just starting to toy with this. After I click submit on my form, I see this message on a white screen: HTTP/1.1 200 OK Server: Is-Processed: true Content-Type: text/html Transfer-Encoding: chunked Date: Tue, 19 Jun 2007 05:14:16 GMT
I’m then taken to my end URL…any suggestion as to what I might have botched (i.e. I’d like to avoid the intermediary white screen with text)?
Scott Hemmeter Said,
June 19, 2007 @ 8:13 am
Hmmmm… I did not test this on multiple different configurations. I was seeing this message before I put the / on the end of the URL the form should post to (see the first comment on the post). However, when it happened, it never took me to the return page. Now that it’s working, I am just getting redirected to the SFDC page (in firefox)
I JUST tested this on IE for the first time and I am seeing what you are seeing. I’ll look into it and see what I can find out.
bigbrownboy Said,
June 19, 2007 @ 10:00 am
Yeah, I just tested in FireFox for the first time and it works a charm. I’ll stay tuned to see if those much smarter than me have an idea for IE.
p.s. new with Salesforce.com at a non-profit and your blog has been a fantastic resource!
Scott Hemmeter Said,
June 19, 2007 @ 10:23 am
I fixed it. You can either download the files again and start over or delete the following line from your index.php file.
curl_setopt($ch, CURLOPT_HEADER, 1);
The only thing that changed in the zip was the index.php file. The line above was removed and so was its associated comment.
bigbrownboy Said,
June 19, 2007 @ 11:20 am
Genius…works nicely. I’ll report back down the road with how its working in practice.
Sundeep Parsa Said,
June 26, 2007 @ 12:01 am
I would like to try this tool for our site. Does anybody know what types of validations are performed by this tool? Do they look at fields like name, email, postal address etc. to keep SPAM out?
Scott Hemmeter Said,
June 26, 2007 @ 8:43 am
Sundeep:
In short, the scripts send everything in your web to lead form to Akismet. You have the ability to tell the script what not to send, if you wish. However, it starts by sending everything.
The tool treats a web to lead request like a blog comment (like the one you submitted on this post). It uses the First Name and Last Name for the Name, the Email as the Email Address and the Website as the URL. It then includes everything (including First Name, Last Name, Email and Website) as the Comment text. It excludes any fields you specify in the “$Akismet_noPass” array of the constants.php file. If you don’t touch it, then it sends (almost) everything. It absolutely sends 100% of the user submitted fields.
Protect Salesforce « Akismet Said,
July 8, 2007 @ 3:31 pm
[...] Read the rest and download the script. [...]
Kumar Said,
July 31, 2007 @ 1:54 pm
I just modified the constants.php and uploaded to another directory. Then, I pointed that directory index.php file in the web to lead form.
Then, when I execute the web to lead form, a blank white page is showing. It is not redirecting back to the retURL page. Please let me know what seems to be the issue.
Thanks.
Scott Hemmeter Said,
July 31, 2007 @ 3:11 pm
@Kumar
All 3 files need to be included in the same directory on your web server. It sounds like you moved one somewhere else. If you move the constants.php file to another directory, then you need to modify the “include” line in the PHP code for that file. Easiest thing to do is to keep them all in the same directory.
When you run it, is the lead at least going into Salesforce.com? Is the redirect the only problem?
Scott Hemmeter Said,
September 11, 2007 @ 5:06 pm
The files were updated with 1 minor change that has a big impact. The web to lead URL was changed to use https.
To fix it in your scripts, either download the new version or search index.php for “http://www.salesforce.com/servlet/servlet.WebToLead?encoding=UTF-8″ and replace it with “https://www.salesforce.com/servlet/servlet.WebToLead?encoding=UTF-8″
ttboy Said,
October 8, 2007 @ 9:02 am
Hi,
I’m testing this script now & it seems to be working well, but I’m having trouble developing a validation rule in Salesforce for this. Would it be possible to post an example?
Thanks so much!
-tt
Scott Hemmeter Said,
October 8, 2007 @ 9:21 am
@ttboy:
The validation rule needs to evaluate to TRUE to stop a Lead from coming in. Since the field you setup in Step #1 of this post is a checkbox, all you need to do is add a reference to the field as your Validation Rule formula. For example, my field is called Akismet_Spam__c.
In my validation rule, I just selected that field from the Field Picker. Thus, my validation rule formula is:
Akismet_Spam__cTry that and test it in the UI first.
Jenn Said,
October 9, 2007 @ 3:31 pm
I keep getting this error:
Parse error: parse error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or ‘}’ in /home/virtual/site201/fst/var/www/html/staging.borregosolar.com/solar-energy-company/contact/web2lead/Akismet.class.php on line 54
Any thoughts?
Scott Hemmeter Said,
October 9, 2007 @ 3:53 pm
@Jenn, make sure you are running PHP 5, not PHP 4. I received this same error once when I was running on PHP4.
Sebastien Said,
October 10, 2007 @ 2:29 pm
Thanks for this excellent solution Scott. Incorporating Akismet in the web-to-lead process is a bright idea.
Spamming is a major problem when implementing a web-to-lead solution, and unfortunately implementing an effective web-to-lead process in Salesforce can turn into a real nightmare as its standard web-to-lead utility not only is targeted by spam, but also by lead duplication.
The solution you’ve developed using Akismet is great, but for those that don’t have sufficient resources (time, programming knowledge…), and would like to easily fill the gap between their website and Salesforce without having to worry about spam, I think a excellent solution is to use FormVester (available from the AppExchange).
FormVester generate leads from any of your existing online forms, without the hassle of reprogramming them…and it is spam-free. The reason is that, it works by adding a snippet tracking code into your website pages (that will execute a hosted script), so that, no OID number is ever exposed in the source code of the page (as opposed to the standart Salesforce web2lead page). Spam bots are then not able to take advantage of this number ensuring a clean spam-free lead generation.
An interesting other thing about FormVester, as opposed to the default Salesforce Web-to-Lead utility, is that it always checks if a lead already exists in Salesforce before creating/updating it (using a 6 rules based filter) so that it will never duplicate the lead.
Give it a try! FormVester for AppExchange
Hope it will help!
ttboy Said,
October 12, 2007 @ 8:25 am
Hi Scott,
Thanks for being so responsive! I’ve uploaded the files to my webserver & am testing now. I’m getting a white screen now after submitting the form - it seems to go to the index.php folder & get stuck. I looked at the comments here & noticed that others were having the same issue. I tried the fixes mentioned, but nothing seemed to work. Let me know if you know of any workarounds.
Best,
t
Scott Hemmeter Said,
October 12, 2007 @ 8:52 am
@ttboy: Is the lead data going into Salesforce.com? If so, do you have a return URL in your web to lead form? It should bring you back there. Also, your HTML form should either point to the folder (http://www.example.com/spamcheck/) with the slash on the end of the URL or you can post directly to index.php (http://www.example.com/spamcheck/index.php). The second way would be the sure fire way to post to the right place. The first way requires your web server to use index.php as a default page when navigating to a folder.
If you give me the URL, I can test it.
ttboy Said,
October 12, 2007 @ 9:23 am
Hi Scott,
I do have a return url. I tried pointing the form to bother to both foldername/ & foldername/index.php. I’ll double check that I’m not missing something….
thanks again!!
-t
How to create leads using PHP and Salesforce SOAP API - Mike Simonds Said,
March 5, 2008 @ 6:22 am
[...] Skype also have you thought of using a web to lead to populate your salesforce leads tables > Fight Web to Lead Spam w/ Akismet | Perspectives on Salesforce.com Anyway get back to me and I hope this helps [...]
Carolyn Said,
March 10, 2008 @ 2:34 pm
Has anyone developed a solution that will work with PHP4? I tried just using Bret Kuhns PHP4 toolkit instead, but no leads appear in Salesforce… we’re a nonprofit implementing a new site with Joomla; I have the form in a wrapper because I couldn’t get it to appear correctly any other way. I expect that we will have spam problems unless I implement something like this, and we can’t afford anything that requires a subscription. Any ideas? Thanks!
Scott Hemmeter Said,
March 10, 2008 @ 3:04 pm
@Carolyn:
Do you have a PHP5 server anywhere in your organization that could host the existing code? That will be your easiest option.
I haven’t implemented it in PHP4. You did the right thing by trying the Akismet PHP4 adapter in place of the one I used. Did you go through the code and change the places using the PHP5 class and re-write them for the PHP4 one? Do you have cURL installed?
Carolyn Said,
March 10, 2008 @ 3:47 pm
Thanks for the quick reply! We don’t host our own website; I only got as far as I did by carefully following your directions, so although I assume cURL is installed, I don’t actually know. Also I don’t know how to rewrite the code for the PHP4 class. (I’m really helpful, I know. NOT.)
So I won’t be surprised to have reached a dead end here, but *if* some helpful soul out there has already worked through the problem…
Jzilla Said,
March 17, 2008 @ 5:21 pm
Hi.
Was there ever any resolution to the retURL redirect issue Kumar and ttboy were having? I’m having the same issue. Lead data is entering sforce and appears to be working otherwise, but no redirect to the retURL; it just sits on the blank page index.php.
I’ve checked the items you mentioned previously, but still to no avail.
Any info appreciated.
Scott Hemmeter Said,
March 17, 2008 @ 8:45 pm
@Jzilla:I have not been able to re-produce the issue you are experiencing. One thing you could try is to add the following lines of PHP code to the beginning of the index.php script.
error_reporting(E_ALL);ini_set('display_errors', '1');This will turn on error handling.
Jzilla Said,
March 23, 2008 @ 9:16 pm
Thanks for the reply Scott.
I tried adding the error_reporting code, but no errors show up. Still just the blank page. I tried just adding phpinfo(); to see what’s running, and it’s reporting PHP 5.2.3, with cURL enabled (cURL Information: libcurl/7.15.3 OpenSSL/0.9.7a zlib/1.2.3 libidn/0.5.6).
I checked the php.ini file and it has register_globals set to Off - might that be an issue? As well as some Zend extensions, nothing else.
Any other ideas? Is there something I should ask the webhost?
In the meantime, as a temporary measure, I created a return.php file with a little cURL header redirect function in it to redirect to the retURL address, adding include(’return.php’); to the index.php file (I couldn’t get it to work otherwise).
It’s not particularly elegant, but basically works, though it ends up “calling up” the retURL page within the index.php script and folder (presumably because of the include), instead of actually redirecting to the .html page. I’d rather it just go directly to the html confirmation page, but haven’t been able to figure it out. I have little-to-none PHP skills.
Scott Hemmeter Said,
March 24, 2008 @ 8:26 am
@Jzilla:
register_globals can be off. It’s off by default in PHP5, I believe.
I did a bit of digging into the code and the Akismet PHP5 class I am using uses PHP’s remote file functions, which requires allow_url_fopen to be ON in your PHP setup. I think there was a recent issue with this setting and many web hosts (including mine) turned this off by default to protect their customers. However, with my web host, I can edit the PHP.ini file and turn it back on. Check to see if allow_url_fopen is ON. If not, see if you can turn it on via your php.ini. I don’t think it’s a major issue to turn on (do your own research), but web hosts were turning it off by default to protect the masses.
Jzilla Said,
March 24, 2008 @ 4:51 pm
According to my phpinfo allow_url_fopen is ON. I don’t know if it matters, but allow_url_include is OFF.
I’m curious if Kumar or ttboy ever resolved their issues.
Either way, thanks for the follow-up.
Carlos Said,
March 25, 2008 @ 5:45 am
We feed our web2lead to Ringlead’s declone before it is routed to sfdc. Will this script work in this scenario?
Scott Hemmeter Said,
March 25, 2008 @ 7:41 am
@Carlos:
If RingLead posts it’s data directly to Salesforce’s web to lead rather than connecting via the API, then this might work. I don’t know anything about RingLead, so I can’t say for sure. The script is expecting a set of data that comes through with the field names as used by Web-To-Lead.
I’d be happy to discuss with you and the folks at RingLead if you help facilitate the connection. You already have my email, so let me know.
Ejay Queja Said,
May 5, 2008 @ 9:08 pm
I know this is out of the topic but ii hope you can help a noob on the web to lead process. Is there a way wherein i can just update the fields via web to lead if the lead has an existing record in the lead tab?
do direct me to the right direction please.
thanks a lot.
Web to lead help - Mike Simonds Said,
May 6, 2008 @ 6:50 am
[...] function from the PHPToolKit. OR You could use this system that is free and can be downloaded here If you have to write a form and capture them it can be done quite easily Do you know PHP? If you [...]
Scott Hemmeter Said,
May 6, 2008 @ 8:06 am
@Ejay Queja:
Doing this will require something custom. You can look into 3rd party application on AppExchange or write something. If you write something, what you’ll want to do post your form to that page (similar to how I do it on this post). Different, though, than the script in this post, you’ll need to have that page perform lookup(s) in Salesforce to match an existing lead and, if found, update it. If not found, then you could create the Lead via the API or send it through to Web-to-Lead.
If you want consulting help with this, please complete the Contact Us form on this site and I’ll see what I can do.
X-Squared On Demand Said,
July 2, 2008 @ 8:47 pm
Wordpress, Akismet, and Web-to-Salesforce…
Scott Hemmeter posted a blog entry about using Akismet and PHP to check Web-to-Lead submissions for spam. We are going to start an experiment using Akismet in the following scenarios:
-Web-to-Lead with Google AdWords enabled
-Web-to-Case
……