July 2015 | Bravehartk2's Blog

Nice Python website crawler framework

22nd July 20154th August 2015 Bravehartk2No Comments

Today I stumbled over http://scrapy.org/ while searching for an OpenSource website crawler. Its an interesting crawling and scraping framework for Python. It looks very convenient and easy to use. The most interesting feature seems to be the possibility to select website elements (f.e. hyperlinks) via CSS-selectors. In any case I’ll give it a try.

Flatten directory structure

18th July 20153rd October 2015 Bravehartk2No Comments

Recently I brought a mp3 player that wasn’t able to shuffle the track through all folders. That’s why I had to bring all tracks to the root directory of the player (flatten directory structure). I did it with the following command, that flattens the whole structure by searching all files in the structure and copying […]

Resolve problems with authorized_keys permissions

18th July 2015 Bravehartk2No Comments

Sometimes I had problems to connect to a server via public key authentication. In most cases I could solve them by setting up the right permissions for the file and path of the authorized_keys file. This can be done as shown below:

$> chmod 700 $HOME/.ssh
$> chmod 600 $HOME/.ssh/authorized_keys
$> chmod go-w $HOME $HOME/.ssh
$> chown `whoami` $HOME/.ssh/authorized_keys

$> chmod 700 $HOME/.ssh

$> chmod 600 $HOME/.ssh/authorized_keys

$> chmod go-w $HOME $HOME/.ssh

$> chown `whoami` $HOME/.ssh/authorized_keys

Get current user, mysql version and other status information

15th July 2015 Bravehartk2No Comments

To get information about the currently connected user and more MySQL status information, you can call the following command if you are already logged in:

mysql> STATUS;

1	mysql> STATUS;

This will give you an output like that:

--------------
mysql  Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1

Connection id:		1044395
Current database:	
Current user:		niceguy@localhost
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.1.73-log Source distribution
Protocol version:	10
Connection:		Localhost via UNIX socket
Server characterset:	latin1
Db     characterset:	latin1
Client characterset:	latin1
Conn.  characterset:	latin1
UNIX socket:		/var/lib/mysql/mysql.sock
Uptime:			2 days 6 hours 26 min 44 sec

Threads: 8  Questions: 399680642  Slow queries: 20292  Opens: 9697  Flush tables: 1  Open tables: 4096  Queries per second avg: 2039.145
--------------

--------------

mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1

Connection id: 1044395

Current database:

Current user: niceguy@localhost

SSL: Not in use

Current pager: stdout

Using outfile: ''

Using delimiter: ;

Server version: 5.1.73-log Source distribution

Protocol version: 10

Connection: Localhost via UNIX socket

Server characterset: latin1

Db characterset: latin1

Client characterset: latin1

Conn. characterset: latin1

UNIX socket: /var/lib/mysql/mysql.sock

Uptime: 2 days 6 hours 26 min 44 sec

Threads: 8 Questions: 399680642 Slow queries: 20292 Opens: 9697 Flush tables: 1 Open tables: 4096 Queries per second avg: 2039.145

--------------

As you can see, you got a lot more information like operation system data, MysSQL version or the number of slow […]

Simple rsync based backup script

11th July 20154th March 2018 Bravehartk2No Comments

This simple bash script is able to backup all given files and folders to a backup destination. I use it to backup all my Raspberry Pis, Banana Pis and also my notebook to my Synology NAS.

#!/bin/bash
# Lockfilecheck for race condition protection 
exec 9>.backupLockFile
if ! flock -n 9 ; then
    echo "Another instance is already running dude!";
    exit 1;
fi
 
#echo "1: $1";
#echo "2: $2";
#echo "3: $3";
#echo "4: $4";
 
DATE=`date +%y%m%d`;
 
SOURCES="/*";
 
if [ "$1" == "help" ]
    then
    echo "Help";
    echo "- USAGE: bash backup_to_synology.sh [DIRTOBACKUP] [EXCLUDEFILEPATH] [BACKUPDESTINATION] [disable-dry-run]";
    echo "- OR: bash backup_to_synology.sh null null [BACKUPDESTINATION] [disable-dry-run]";
    echo "- OR: bash backup_to_synology.sh [DIRTOBACKUP] null [BACKUPDESTINATION] [disable-dry-run]";
    echo "- OR: bash backup_to_synology.sh [DIRTOBACKUP] null [BACKUPDESTINATION] [disable-dry-run] [logfile]";
    exit 0;
elif [ -n "$1" -a "$1" != "null" ]
    then 
    SOURCES="$1";
fi 
 
EXCLUDEFILE="";
if [ -n "$2" -a "$2" != "null" ]
    then
    EXCLUDEFILE="--exclude-from=$2";
fi
 
# Define remote target
TARGET="";
if [ -n "$3" -a "$3" != "null" ] 
    then
    TARGET="$3";
else
    echo "You have to give me a destination";
    exit 1;     
fi
 
DRYRUN="--dry-run";
if [ -n "$4" -a "$4" == "disable-dry-run" ]
    then
    DRYRUN="";
fi
# rsync base params
# --delete - delete destination files that dosn't exist on the source
# --exclude-from - file / folders to ignore can be 
#
# use -–dry-run to simulate backup -> this is the default mode unless disable-dry-run is set
PARAMS="-azvrpuE --delete";
 
# Path to logfile
LOGFILE="rsync_log.$DATE.log";
if [ -n "$5" -a "$5" != "null" ]
    then
    LOGFILE="$5";
fi

# determine the path to "rsync"
RSYNC=`which rsync`;
 
# run backup command
$RSYNC $PARAMS $DRYRUN $EXCLUDEFILE $SOURCES $TARGET > $LOGFILE;
 
exit 0;

#!/bin/bash

# Lockfilecheck for race condition protection

exec 9>.backupLockFile

if ! flock -n 9 ; then

echo "Another instance is already running dude!";

exit 1;

#echo "1: $1";

#echo "2: $2";

#echo "3: $3";

#echo "4: $4";

DATE=`date +%y%m%d`;

SOURCES="/*";

if [ "$1" == "help" ]

then

echo "Help";

echo "- USAGE: bash backup_to_synology.sh [DIRTOBACKUP] [EXCLUDEFILEPATH] [BACKUPDESTINATION] [disable-dry-run]";

echo "- OR: bash backup_to_synology.sh null null [BACKUPDESTINATION] [disable-dry-run]";

echo "- OR: bash backup_to_synology.sh [DIRTOBACKUP] null [BACKUPDESTINATION] [disable-dry-run]";

echo "- OR: bash backup_to_synology.sh [DIRTOBACKUP] null [BACKUPDESTINATION] [disable-dry-run] [logfile]";

exit 0;

elif [ -n "$1" -a "$1" != "null" ]

then

SOURCES="$1";

EXCLUDEFILE="";

if [ -n "$2" -a "$2" != "null" ]

then

EXCLUDEFILE="--exclude-from=$2";

# Define remote target

TARGET="";

if [ -n "$3" -a "$3" != "null" ]

then

TARGET="$3";

else

echo "You have to give me a destination";

exit 1;

DRYRUN="--dry-run";

if [ -n "$4" -a "$4" == "disable-dry-run" ]

then

DRYRUN="";

# rsync base params

# --delete - delete destination files that dosn't exist on the source

# --exclude-from - file / folders to ignore can be

# use -–dry-run to simulate backup -> this is the default mode unless disable-dry-run is set

PARAMS="-azvrpuE --delete";

# Path to logfile

LOGFILE="rsync_log.$DATE.log";

if [ -n "$5" -a "$5" != "null" ]

then

LOGFILE="$5";

# determine the path to "rsync"

RSYNC=`which rsync`;

# run backup command

$RSYNC $PARAMS $DRYRUN $EXCLUDEFILE $SOURCES $TARGET > $LOGFILE;

exit 0;

Example call:

$> bash backup_to_synology.sh "/data/snots/*" exclude.txt backup@ds.intern.ask-sheldon.com:/volume1/NetBackup/thinkpad disable-dry-run logfile.log

1	$> bash backup_to_synology.sh "/data/snots/*" exclude.txt backup@ds.intern.ask-sheldon.com:/volume1/NetBackup/thinkpad disable-dry-run logfile.log

If you want to run it as a crontab, you can use key based authentication (see […]

Create UTF-8 Database

11th July 201512th January 2017 Bravehartk2No Comments

To crate a UTF-8 database:

CREATE DATABASE sns_typo3   DEFAULT CHARACTER SET utf8   DEFAULT COLLATE utf8_general_ci;

1	CREATE DATABASE sns_typo3 DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

Reinstall Magento

11th July 201511th October 2017 Bravehartk2No Comments

To reinstall Magtento with the installation wizard you have to: delete app/etc/local.xml clear cache

$> rm -rf /path/to/mage/var/cache/*

1	$> rm -rf /path/to/mage/var/cache/*

reload your shop frontend

Recursively set permissions

10th July 20153rd August 2015 Bravehartk21 Comment

To change the permissions of all files and / or folders insight the current folder recursively you can run one of the following command:

$> sudo find . -type d -exec chmod 644 {} \; #all folders get 644
$> sudo find . -exec chmod g+r {} \;         #all files and folders get group read rights
$> sudo find . -type f -exec chmod 755 {} \; #all files get 755

$> sudo find . -type d -exec chmod 644 {} \; #all folders get 644

$> sudo find . -exec chmod g+r {} \; #all files and folders get group read rights

$> sudo find . -type f -exec chmod 755 {} \; #all files get 755

All options can be found on http://unixhelp.ed.ac.uk/CGI/man-cgi?find.

Comments in YAML

4th July 20154th July 2015 Bravehartk2No Comments

It’s no big deal to comment out a line in YAML. Just put a sharp (#) as the first character of that line.

fileSync:
    typo3_user_upload:
        #rsync_params: '--delete -avhHxS'
        #source:   'shared/typomage/fileadmin/user_upload'
        #target:   'shared/typomage/fileadmin/user_upload'
    typo3_redaktion:
        rsync_params: '--delete -avhHxS'
        source:   'shared/typomage/fileadmin/redaktion'
        target:   'shared/typomage/fileadmin/redaktion'
#    magento_media:
#        rsync_params: '--delete -avhHxS'
#        source:   'shared/typomage/media/catalog/product'
#        target:   'shared/typomage/media/catalog/product'

fileSync:

typo3_user_upload:

#rsync_params: '--delete -avhHxS'

#source: 'shared/typomage/fileadmin/user_upload'

#target: 'shared/typomage/fileadmin/user_upload'

typo3_redaktion:

rsync_params: '--delete -avhHxS'

source: 'shared/typomage/fileadmin/redaktion'

target: 'shared/typomage/fileadmin/redaktion'

# magento_media:

# rsync_params: '--delete -avhHxS'

# source: 'shared/typomage/media/catalog/product'

# target: 'shared/typomage/media/catalog/product'

As you can see the lines 3-5 and 10-13 will be interpreted as comments. Unfortunately there is no block comment syntax in YAML (at least I didn’t find any 😉 !).

Bravehartk2's Blog

- next level braining -

Month: July 2015

Nice Python website crawler framework

Flatten directory structure

Resolve problems with authorized_keys permissions

Get current user, mysql version and other status information

Simple rsync based backup script

Create UTF-8 Database

Reinstall Magento

Recursively set permissions

Comments in YAML

July 2015
M	T	W	T	F	S	S
« Jun				Aug »
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this: