Error 403 - Forbidden for url: https://www.craigslist.org/about/sites

Hi Julio,

I have used your code before (early 2020), but now I'm getting the error below when trying to import CraigslistHousing, using "from craigslist import CraigslistHousing":

HTTPError: 403 Client Error: Forbidden for url: https://www.craigslist.org/about/sites

Not sure why, it seems that could be related with this issue: https://stackoverflow.com/questions/16627227/http-error-403-in-python-3-web-scraping.

Do you happen to know why this is happening?

Thanks,

Seems like this works on my end. Did you upgrade python-craigslist to the latest version? I have a feeling this issue might be agnostic of package upgrade, but it doesn't hurt..

Yep, I did the upgrade and continue to have the same issue. Using v1.1.0 and python 3.6, I'm using Google's Colab notebooks.

Ah, this looks to be a problem with the requests library in your environment, not python-craigslist, per se.
I'm guessing the same exception would be thrown if you executed this:

import requests
requests.get("https://www.craigslist.org/about/sites")

You are completely right, I also tried in a new colab and got "<Response [403]>"

If I run the code below I get a successful response and the page code. I believe it's related with the web scraping issue in this page.

from urllib.request import Request, urlopen
req = Request('https://www.craigslist.org/about/sites', headers={'User-Agent': 'XYZ/3.0'})
webpage = urlopen(req, timeout=10).read()

print(webpage)

@luisandrecunha

Thanks for reporting @luisandrecunha.

Interesting. Seems like Craigslist is blocking requests coming from your IP (or Google's Colab IPs). I'm guessing the IP hit a max number of requests per day/hour/minute.

Do you mind running the code suggested by @irahorecka but setting a User-Agent like you did with urllib:

import requests
requests.get("https://www.craigslist.org/about/sites", headers={'User-Agent': 'python-craigslist/1.1.0'})

If this works fine, I'll add a default User-Agent to all requests to prevent this from happening in the future.

Thanks!

@juliomalegria

Hi @juliomalegria ,

It seems that Google's Colab IPs is blocked by Craigslist... I successfully ran the code in a local jupyter notebook and it worked like a charm.

I tried the code you suggested in Colab and continued to get the 403 response... However I receive the right page if I use the code below, not sure if somehow the code could be adapted.

from urllib.request import Request, urlopen
req = Request('https://www.craigslist.org/about/sites', headers={'User-Agent': 'XYZ/3.0'})
webpage = urlopen(req, timeout=10).read()

print(webpage)

Thank you again,

Just a heads up, I've got the exact same issue. I've been running my code for more than a year and this just happened this week. So, something must have changed on the craigslist side? I'll have to dig into the code. I can cut and paste the url into a browser and it works fine. Just wanted to let you know of another user with the same issues.

>>> import requests
>>> requests.get('https://boston.craigslist.org')
<Response [200]>
>>> requests.get('https://boston.craigslist.org/search')
<Response [403]>
>>> requests.get('https://boston.craigslist.org/search',headers={'User-Agent': 'XYZ/3.0'})
<Response [403]>

I tried it on a couple of computers, so I don't think it's IP related. Guess how the servers are seeing the 'requests' library versus a regular library.

Thanks!

Hey everyone! Sorry for the inactivity. I've released a new version (1.1.1) adding a User-Agent to requests.get. Hopefully that will solve the issue, please report back if it does or doesn't. If it doesn't I'll have to change libraries to urllib.
Thanks!

I am still getting the 403 error with the updated utils.py.

+1 Having the same behavior - 403s on /search paths through just a general requests.get() call so the library/class is also not functioning.

Also note I tried taking the headers object from the cURL to /search which loads in a regular browser and used that for the requests call which they also blocked.

I used a selenium driver I had with some mods I've used in the past and I was able to load /search just fine so I don't suspect they are doing something super sophisticated to block the request.

Okay I've dug into it a bit more - I don't think this has anything do to with user agents or anything they are blocking like that. I recommend upgrading both the requests and urlib3 library pip install urllib3 --upgrade pip install requests --upgrade. Once I did that things started working again. So not sure the actual issue - as older versions of those libraries were working - but with the updates it looks fine to me.

After I did that I tested the request function (which is effectively requests.get()) works:

import requests
import urllib3
from craigslist import utils

>> requests.__version__
Out[5]: '2.25.1'

>>urllib3.__version__
Out[6]: '1.26.3'

>> utils.requests_get('https://boston.craigslist.org/search')
Out[8]: <Response [200]>

@KeeonTabrizi

Thanks @KeeonTabrizi! That's a very good point.
I've updated the requirements to include some minimum version for requirements (requests and beautifulsoup4).
Can anyone having issues try updating their library (pip install python-craigslist --upgrade) and let me know if this fixed the issue.
Thanks again!

Hey guys. I am not a power user, but I have found that the latest *idna* version is incompatible with *requests*. If you installed the latest *idna* then just run *requests* upgrade and it will revert the *idna* version. I have no clue that it could be your troubles, but it could be a factor. Hope this helps. Le mar. 23 févr. 2021 à 13:15, Julio M. Alegria <notifications@github.com> a écrit :

…

Thanks @KeeonTabrizi <https://github.com/KeeonTabrizi>! That's a very good point. I've updated the requirements to include some minimum version for requirements (requests and beautifulsoup4). Can anyone having issues try updating their library (pip install python-craigslist --upgrade) and let me know if this fixed the issue. Thanks again! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXNCUNQTBYAWRMKIOJJWQDTAOL47ANCNFSM4VVVT3VQ> .

Hey y'all, thanks so much for taking the time to fix this! So, it could just be how my packages were managed, but, when I performed (pip install python-craigslist --upgrade) it updated requests but not urllib3. I guess urllib3 is used by requests. So, it did not work with just upgrading python-craigslist. But, after updating both request and urllib3 to the latest, back up and running! Maybe consider adding urllib to the requirements? Thanks again!!

These versions are what got my code working:

>>> requests.__version__
'2.25.1'
>>> urllib3.__version__
'1.26.3'

PS. great module, it's helped me get some great deals on Craiglist!

Hey y'all, thanks so much for taking the time to fix this! So, it could just be how my packages were managed, but, when I performed (pip install python-craigslist --upgrade) it updated requests but not urllib3. I guess urllib3 is used by requests. So, it did not work with just upgrading python-craigslist. But, after updating both request and urllib3 to the latest, back up and running! Maybe consider adding urllib to the requirements? Thanks again!!

These versions are what got my code working:
>>> requests.__version__
'2.25.1'
>>> urllib3.__version__
'1.26.3'
PS. great module, it's helped me get some great deals on Craiglist!

+1 this fixed everything. Good catch!

@cwittwer

@cwittwer, @jraVette, @usctzen, @KeeonTabrizi, @luisandrecunha If you guys are interested in a new Craigslist API format, check out pycraigslist.
I enjoy python-craigslist, but there were some features I wanted to implement immediately. Some additional features are in the works.

Thanks, I'll check it out. Le mar. 30 mars 2021 à 18:42, Ira Horecka ***@***.***> a écrit :

…

@cwittwer <https://github.com/cwittwer>, @jraVette <https://github.com/jraVette>, @usctzen <https://github.com/usctzen>, @KeeonTabrizi <https://github.com/KeeonTabrizi>, @luisandrecunha <https://github.com/luisandrecunha> If you guys are interested in a new Craigslist API format, check out pycraigslist <https://github.com/irahorecka/pycraigslist>. I enjoy python-craigslist, but there were some features I wanted to implement immediately. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXNCULAQFAEV7YGYKK2MNDTGH5OTANCNFSM4VVVT3VQ> .

Ira, Just gave it a quick try and I am getting an error. The script finds the forsale.mca but does not recognize the forsale.mcy mca is motorcycle all and mcy is motorcycles by owner. *Traceback (most recent call last): File "C:/Users/mgpd/PycharmProjects/molivo/py_clist.py", line 3, in <module> print(pycraigslist.forsale.mcy.get_filters())AttributeError: type object 'forsale' has no attribute 'mcy'* Marc @usctzen Le mar. 30 mars 2021 à 18:42, Ira Horecka ***@***.***> a écrit :

…

@cwittwer <https://github.com/cwittwer>, @jraVette <https://github.com/jraVette>, @usctzen <https://github.com/usctzen>, @KeeonTabrizi <https://github.com/KeeonTabrizi>, @luisandrecunha <https://github.com/luisandrecunha> If you guys are interested in a new Craigslist API format, check out pycraigslist <https://github.com/irahorecka/pycraigslist>. I enjoy python-craigslist, but there were some features I wanted to implement immediately. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXNCULAQFAEV7YGYKK2MNDTGH5OTANCNFSM4VVVT3VQ> .

@usctzen

Hey @usctzen, I always appreciate your feedback. Could you post the same issue in pycraigslist issues? I’ll address it there :)

Sure thing! Le mar. 30 mars 2021 à 21:36, Ira Horecka ***@***.***> a écrit :

…

Hey @usctzen <https://github.com/usctzen>, I always appreciate your feedback. Could you post the same issue in pycraigslist issues? I’ll address it there :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXNCUOF36FRDJWJCHYHW23TGIR3FANCNFSM4VVVT3VQ> .

Hey everyone! Sorry for the delay, I've updated the requirements in 88a6b73 and pushed a new version in PyPI. Could anyone confirm if the issue is fixed with this?
Thanks for all the patience!

I am still having this issue

Navigation Menu

Repository navigation

Error 403 - Forbidden for url: https://www.craigslist.org/about/sites #105

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Error 403 - Forbidden for url: https://www.craigslist.org/about/sites #105

Description

Activity

irahorecka commented on Jan 5, 2021

luisandrecunha commented on Jan 5, 2021

irahorecka commented on Jan 5, 2021

luisandrecunha commented on Jan 5, 2021

juliomalegria commented on Jan 7, 2021

luisandrecunha commented on Jan 8, 2021

jraVette commented on Feb 18, 2021

juliomalegria commented on Feb 19, 2021

cwittwer commented on Feb 20, 2021

KeeonTabrizi commented on Feb 21, 2021

KeeonTabrizi commented on Feb 22, 2021

juliomalegria commented on Feb 23, 2021

usctzen commented on Feb 23, 2021

jraVette commented on Feb 23, 2021

cwittwer commented on Mar 11, 2021

irahorecka commented on Mar 30, 2021

usctzen commented on Mar 30, 2021

usctzen commented on Mar 30, 2021

irahorecka commented on Mar 30, 2021

usctzen commented on Mar 30, 2021

juliomalegria commented on Apr 6, 2021

Agwebberley commented on Nov 21, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions