Jekyll2024-09-24T15:02:07+00:00https://teagom.github.io/feed.xmlTiago SMBookmark this to keep update!Tiago de Souza MoraesCounting number of times each IP address appears in log file.2024-09-24T00:00:00+00:002024-09-24T00:00:00+00:00https://teagom.github.io/2024/09/24/counting-number-of-times-each-IP-address-appears-in-log-fileHow to print a total of hits per IP?

I will use the common log of apache2

/var/log/apache2/access.log

This is the format of access.log, I want to print total of hits per IP.

192.168.0.x - - [24/Sep/2024:02:03:55 -0300] "GET / HTTP/1.1" 200

To explain parts of last command line

1 print only IP column

cd /var/log/apache2
more access.log | awk '{print $1}'
# output
89.208.11.10
45.148.10.242
95.214.55.138
45.148.10.242

2 sort IP list

more access.log | awk '{print $1}' | sort
# output
89.208.11.y
89.208.11.y
89.208.11.y
95.214.55.x
95.214.55.x
95.214.55.x
95.214.55.x

3 counter total of hits per IP

more access.log | awk '{print $1}' | sort | uniq -c
# output
 1 141.98.11.122
 1 178.211.139.188
 2 185.16.39.118
 1 185.224.128.47
 1 35.177.209.183
98 45.148.10.242
 1 45.84.89.2
 1 46.174.191.30
 1 52.228.160.228
 2 5.8.11.202
 3 65.49.20.67
 1 79.124.59.226
 1 79.124.8.107
 1 81.17.22.122
 1 82.157.247.165
 2 89.190.156.137
10 89.208.11.10
30 95.214.55.138
 2 95.214.55.43

Custom filter

Filter 40x apache2 error, counter total of hits per IP.

more access.log | grep '\" 40[0|1|2|3|4]' | awk '{print $1}' | sort | uniq -c

Filter 200 apache2 success, counter total of hits per IP.

more access.log | grep '\" 200' | awk '{print $1}' | sort | uniq -c
]]>Tiago de Souza MoraesBasic commands and tools2024-09-19T00:00:00+00:002024-09-19T00:00:00+00:00https://teagom.github.io/2024/09/19/basic-commands-toolsMerge PDF files 
gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -dAutoRotatePages=/None -sOutputFile=finished.pdf  rg2020.pdf rg2020-ver.pdf

How to print a total of register from a table? Total COUNT in postgreSQL.

To create a file with the follow content:

echo "SELECT COUNT(*) FROM client_client WHERE confirmed=1;"  > /tmp/command.sql

Print date time and total of occurences 5 to 5 minutes, sleep command work’s in seconds, crtl+c to out. Execute the command as postgres user.

su - postgres 
while [ -z ]; do date && psql -d mydatabase < command.sql; sleep 300; done

The best traffic monitoring

iptraf-ng

My current IP address via line command

curl checkip.amazonaws.com
curl ifconfig.me
curl icanhazip.com
curl ipecho.net/plain
curl ifconfig.co

Show resume of process and use memory for each service

https://github.com/pixelb/ps_mem
git clone https://github.com/pixelb/ps_mem.git
]]>Tiago de Souza MoraesHow to restrict commands for a SSH connection use authorized key?2024-09-19T00:00:00+00:002024-09-19T00:00:00+00:00https://teagom.github.io/2024/09/19/how-to-restrict-commands-for-a-ssh-connection-use-authorized-keyIn this case developers can only run git commands!

git commit
git pull
git clone
git push

Developer computer is authorized for ssh connection but allow only git commands, any other bash/sh command will be dropped. In order to limit the user to a single command, the parameter command = is entered before the key.

To create this file /home/user/.ssh/gitcommads, content:

#!/bin/sh
exec git-shell -c "$SSH_ORIGINAL_COMMAND"

Set permission do exec

chmod +x /home/user/.ssh/gitcommads

To edit /home/user/.ssh/authorized_keys and add commands before pub key.

# admin
ssh-rsa this-my-priv-key-admin.....
# dev 1
command="~/.ssh/gitcommands",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa this-my-priv-key-C00001......
# dev 2
command="~/.ssh/gitcommands",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa this-my-priv-key-C00002......
# dev 3
command="~/.ssh/gitcommands",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa this-my-priv-key-C00003......

To test you have to use ssh from developer computer to test!

git clone ssh://login@server:/path
]]>Tiago de Souza MoraesDocs2024-09-19T00:00:00+00:002024-09-19T00:00:00+00:00https://teagom.github.io/2024/09/19/links-to-docsMonitoring process, cpu e memory

https://www.linuxtechi.com/generate-cpu-memory-io-report-sar-command/ https://www.thegeekstuff.com/2011/03/sar-examples/ https://www.thegeekstuff.com/2014/11/pidstat-examples/ https://www.tecmint.com/sysstat-commands-to-monitor-linux/ https://promet.github.io/2009/06/tracing-memory-leaks-with-pidstat/

AWS cloudWatch custom log

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/service_code_examples.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/ExtractBytesExample.html https://docs.aws.amazon.com/pt_br/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html

Apache2

https://linuxbeast.com/tutorials/aws/how-to-export-logs-from-apache2-web-server-to-amazon-cloudwatch/ https://aws.amazon.com/blogs/mt/simplifying-apache-server-logs-with-amazon-cloudwatch-logs-insights/

]]>Tiago de Souza MoraesPFSense commands2024-09-19T00:00:00+00:002024-09-19T00:00:00+00:00https://teagom.github.io/2024/09/19/pfsense-commandsEasy and direct commands for PFSense and FreeBSD

The following command adds a firewall rule, allowing tcp traffic in on port 443 from remote IP XX.XX.XX.XX: to the WAN IP on YY.YY.YY.YY:

easyrule pass wan tcp XX.XX.XX.XX YY.YY.YY.YY 443

You can also allow SSH access and set up a remote port forward (ssh -L localport:remoteip:remoteport remoteip):

easyrule pass wan tcp XX.XX.XX.XX YY.YY.YY.YY 22

Traffic network

iftop -i re0 -F -P -o 2s

Nginx don’t start, service nginx one restart

erro

Performing sanity check on nginx configuration:
nginx: [emerg] cannot load certificate key "/usr/local/etc/nginx/cert.key": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/local/etc/nginx/cert.key','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

to fix

cd /usr/local/etc/nginx
openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout cert.key -out cert.pem -days 3650
]]>Tiago de Souza MoraesFacebook bots flooding a web server, how to block?2024-08-08T00:00:00+00:002024-08-08T00:00:00+00:00https://teagom.github.io/2024/08/08/facebook-bots-hexcodeA excessive / massive requests from facebook are flooding a web (apache2) server and freeze it for each 30 minutes.

Doubts

  1. Why hex code found in the URL? Exploit memory address?
  2. Why massive and excessive requests?
  3. Are facebook-bots or attacker are using the game/app dev environment of facebook to do this?
  4. Bad robot?

/var/log/apache2/access.log

173.252.107.x - - [01/Aug/2024:10:20:45 -0300] "GET
index?q=\xe7\xbb\x8d\xe5\x85\xb4\xe5\x93\xaa\xe9\x87\x8c\xe6\x9c\x89\xe6\x9c
HTTP/1.1" 200 3847 "-" "facebookexternalhit/1.1
(+http://www.facebook.com/externalhit_uatext.php)"

Count hits daily of apache access log

10.491 28/Jul/2024 without rule
 8.859 29/Jul/2024 without rule
 9.340 30/Jul/2024 without rule
 3.573 31/Jul/2024 without rule
 1.238 01/Aug/2024 with rule, 237 blocked IPs from 12h00 until 17h25.

To block, add this rule in the fail2ban service.

/etc/fail2ban/filter.d/apache-facebook-hexcode.conf

# Generic configuration items (to be used as interpolations) in other
# apache filters.
[Definition]
datepattern = ^[^\[]*\[({DATE})
failregex   = ^<HOST> -.*(GET|POST|HEAD).*\\x[a-zA-Z0-9].*$
              ^<HOST> -.*facebookexternalhit.*$
              
ignoreregex =
]]>Tiago de Souza MoraesInstalando o PHL 8.22024-07-19T00:00:00+00:002024-07-19T00:00:00+00:00https://teagom.github.io/2024/07/19/instalando-phl-82Instalando e configurando sistema PHL - Personal Home Library

Autor e download do PHL - http://www.elysio.com.br

BibliotecaPHL é um sistema para gerenciar uma biblioteca, empréstimo, consulta, cadastros e tudo aquilo que uma biblioteca necessita para funcionar. Aqui mostro como instalar o PHL, não tenho conhecimento de como gerenciar o sistema, banco de dados, usuários, livros e outros.

Todos os passos a seguir deve ser feito com usuário ROOT ou ter acesso sudo.

Não funciona em 64Bits porque arquivos foram compilados em 32bits. As informações inseridas em arquiterura 64 ficam todas bagunçadas, sem nenhum sentido de leitura.

Ambiente

Linux Debian Lenny Arquitetura 32Bits ou 686.
Apache2 e PHL82

Diretório que vamos usar para descompactar o tar.gz do PHL
/usr/local/src/

Diretorio padrao Apache2, Document Root  
/var/www

Diretorio CGI do apache2
/usr/lib/cgi/bin

Diretorio do PHL dentro do apache2
/var/www/http

Download http://www.elysio.com.br/site/downloads.html

Instalar o apache2 e ligar/carregar o modulo CGI.

apt-get update
apt-get install apache2
a2enmod cgid

Pacotes instalados no sistema, pode usar o comando:

dpkg -l | grep -i apache
ii  apache2 2.2.11-2ubuntu2.5   Apache HTTP Server metapackage
ii  apache2-mpm-prefork   2.2.11-2ubuntu2.5   Apache HTTP Server - traditional non-threade
ii  apache2-utils 2.2.11-2ubuntu2.5  utility programs for webservers
ii  apache2.2-common    2.2.11-2ubuntu2.5    Apache HTTP Server common files
ii  libapache2-mod-php5    5.2.6.dfsg.1-3ubuntu4.4 server-side, HTML-embedded scripting languag
ii  libapr1    1.2.12-5ubuntu0.1    The Apache Portable Runtime Library
ii  libaprutil1    1.2.12+dfsg-8ubuntu0.3   The Apache Portable Runtime Utility Library

Instalação

vamos para o diretorio src

cd /usr/local/src

Faça o download do pacote mais novo no site do fabricante com wget

wget -c http://www.elysio.com.br/downloads/phl82_090619.tar.gz

vamos descompactar o pacote:

tar zxfv phl82_090619.tar.gz

um diretorio “http” foi criado, é o conteudo do PHL. Vamos copia-lô para o diretorio www do apache para que fique acessivel pelo navegador.

cp /usr/local/src/http /var/www/. -prav
cd /var/www/http

Inicie o apache2

/etc/init.d/apache2 start

Vamos editar o arquivo cgi-bin/phl82.cip para alterar os caminhos dos arquivos do PHL, vamos colocar o caminho completo nas configurações.Para isso vamos usar o comando sed.Os camandos abaixo fazem essas alterações.

Faça uma copia do arquivo phl82.cip

cp -prav cgi-bin/phl82.cip cgi-bin/phl82.cip.original

Alterando o original para o caminho do diretorio apache.

more cgi-bin/phl82.cip.original | sed s/http/'var\/www\/http'/g > cgi-bin/phl82.cip

O conteudo original do arquivo

phl_*=/http/bases/phl_*
actab=/http/bases/actab
uctab=/http/bases/uctab
menu*=/http/www/phl82/html/menu*
cabe*=/http/www/phl82/html/cabe*
mens*=/http/www/phl82/html/mens*
rest*=/http/www/phl82/html/rest*
inde*=/http/www/phl82/html/inde*
logo*=/http/www/phl82/html/logo*
atra*=/http/www/phl82/php/mail_lote/atra*
aler*=/http/www/phl82/php/mail_lote/aler*
disp*=/http/www/phl82/php/mail_lote/disp*
usua*=/http/www/phl82/php/mail_lote/usua*
phl.css=/http/www/phl82/css/phl.css
tab_*=/http/cgi-bin/phl82/tabs/tab_*

como deve ficar

00*=/var/www/http/bases/00*
phl_*=/var/www/http/bases/phl_* actab=/var/www/http/bases/actab
uctab=/var/www/http/bases/uctab
menu*=/var/www/http/www/phl82/html/menu*
cabe*=/var/www/http/www/phl82/html/cabe*
mens*=/var/www/http/www/phl82/html/mens*
rest*=/var/www/http/www/phl82/html/rest*
inde*=/var/www/http/www/phl82/html/inde*
logo*=/var/www/http/www/phl82/html/logo*
atra*=/var/www/http/www/phl82/php/mail_lote/atra*
aler*=/var/www/http/www/phl82/php/mail_lote/aler*
disp*=/var/www/http/www/phl82/php/mail_lote/disp*
usua*=/var/www/http/www/phl82/php/mail_lote/usua*
phl.css=/var/www/http/www/phl82/css/phl.css
tab_*=/var/www/http/cgi-bin/phl82/tabs/tab_*

Verifique se o arquivo foi alterado corretamente:

more cgi-bin/phl82.cip

Feito isso, vamos criar um link do PHL para o diretorio CGI do apache:

ln -s /var/www/http/cgi-bin/* /usr/lib/cgi-bin/.

Vamos criar outro link, do PHL82 para a raiz do diretorio apache, assim deixamos o sistema PHL acessivel pelo navegador.

ln -s /var/www/http/www/phl82 /var/www/.

Permissão para o Apache2

chown www-data.www-data /var/www/http -R

Agora já podemos acessar o PHL pelo navegador, http://ip_servidor/phl82/, se você quer acessar apenas o endereço do servidor e ir direto ao PHL82, faça o link do phl82/index.html para o document root do apache /var/www criando um simbolic link.

ln -s http/www/phl82/index.html /var/www/. -f

Agora é só acessar o http://ip_servidor com o navegador que irá abrir diretamente o PHL82.

Erro e solução

Erro PHL 82 - index.html - valido

Nota sobre copyright contida no arquivo “index.html” foi violada!

Solução

phl82.cip - caminho para diretorio está incorreto, verificar todos ou conteúdo do index.html foi alterado errado.

Erro com Ubuntu 64Bits

Arquivo wxis.exe foi compilado em 32bits.

tail -f /var/log/apache2/error.log 
[Tue Oct 16 13:36:58 2012] [error] (2)No such file or directory: exec of '/var/www/phl82/cgi-bin/wxis.exe' failed
[Tue Oct 16 13:36:58 2012] [error] [client 192.168.250.250] Premature end of script headers: wxis.exe, referer: http://172.0.0.1/phl82/

Erro WXIS|fatal error

WXIS|fatal error|unavoidable|recread/xropn/w| 
usuario apache não tem permissão para ler arquivos

Solução Configurar permissão de dono aos arquivos para o usuario que executa o apache2, normalmente é o user www-data.

cd /var/www/http
chmod 755 bases -R
chown www-data:www-data bases -R
]]>Tiago de Souza MoraesOpenSSH - Custom settings for more security.2024-07-11T00:00:00+00:002024-07-11T00:00:00+00:00https://teagom.github.io/2024/07/11/openssh-server-custom-settingsFor more security change ssh port for a custom port, but a risk to sshd broken and you can lost server ssh access. To test a custom settings before to the production.

  1. How to do this in safe mode?
  2. How to avoid to lost ssh connection?

Enviromment

Linux Ubuntu Server 20.04 64
Required root access / sudo

Make backup of sshd_config file

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.port22
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.port5000

You can run a second instance of openssh-server using a custom configuration, make all change you need.

vim /etc/ssh/sshd_config.port5000
change port to 5000
Port 5000

To test the syntax of custom config file

sshd -t -f /etc/ssh/sshd_config.port5000

Run a new server using new sshd_config, do not forget to open port at firewall.

-4d: Ipv4 + debug
-f : config file

/usr/sbin/sshd -4d -f /etc/ssh/sshd_config.port5000

ctrl+c to stop

Open a new terminal and try a ssh connection using port 5000 from authorized client.

ssh user@server -p 5000

If you have success, then copy ssh_config.port5000 to sshd_config (default) and restart the service. Try lot of times new connection before leaving your server!

cp /etc/ssh/sshd_config.port5000 /etc/ssh/sshd_config
sudo service ssh restart
]]>Tiago de Souza MoraesHow to convert lot of images?2024-07-10T00:00:00+00:002024-07-10T00:00:00+00:00https://teagom.github.io/2024/07/10/convert-lot-of-imagesThis text explain how to convert lof of images of lot of folders.

Enviromment

  1. Linux Ubuntu 20.04 64
  2. convert command, ImageMagick Package

We will to use the convert command, bash, convert - https://packages.ubuntu.com/disco/graphicsmagick-imagemagick-compat Install Image Magick

apt install imagemagick

This is a sample

cd /tmp/my-albuns
ls -la
drwxrwxr-x  2 user user    4096 Out  4 19:10 ci
drwxrwxr-x  2 user user    4096 Jul 12 16:07 eletrica
drwxrwxr-x  2 user user    4096 Out 13 19:11 componentes-diversos

set variables

SIZE='50%'	# resize image to 50% of owner size
OUT='800'	# output to converted folder name, 800x600

What the script will do?

  1. create a folder with name-800
  2. convert all images of folder to 50% of original size

F: Folder name I: Image file

cd /tmp/my-albuns
SIZE='50%'
OUT='800'
for F in $(ls)
	do
	mkdir $F-$OUT
	for I in $(ls $F)
		do
		echo "+ $F/$I";
		convert -resize $SIZE $F/$I $F-$OUT/$I;
	done
done

output

drwxrwxr-x  2 user user    4096 Out  4 19:10 ci
drwxrwxr-x  2 user user    4096 Out  4 19:10 ci-800
drwxrwxr-x  2 user user    4096 Jul 12 16:07 eletrica
drwxrwxr-x  2 user user    4096 Jul 12 16:07 eletrica-800
drwxrwxr-x  2 user user    4096 Out 13 19:11 componentes-diversos
drwxrwxr-x  2 user user    4096 Out 13 19:11 componentes-diversos-800

Be carefull To Delete folder that contain 800 at end of name and all content.

find . -name '*800' -exec rm -rf '{}' \;
]]>Tiago de Souza Moraes