LFCE.5 Linux Service Management - Advanced HTTP Services

Posted Jul 16, 2020 Updated Jan 23, 2024

By Grace L

13 min read

LFCE: Linux Service Management - Advanced HTTP Services

[toc]

build scalable internet architectures

resources consumed: find the bottleneck, mitigate it

hardware

access: spread out the load and minimize user time on server resources

lantency, reply slow down, connection increase, holding resource increase,overall server load increase.
- CPU, disk, memory(memory, system memory), network(high capacity, low lantency)
load balancing
- hardware:
- software: Squid, HAprocy, Nginx
- round robin DNS
SSL offload (increase scalability)
- move encryption of http to a dedicated device
- conserver CPU on primary web server
- common on load balancer or reverse proxy
caching
- pre-staging the resource
- static content
- database content
- cache layer
  - dedicated caching tier
  - internal cache in each web server
  - file system cache
- common caching engines: Squid, HAprocy, Nginx, Apache Traffic Server, HAProxy
compression
- compress the site before send, decompress in client side,
  - reduce bandwich, page load time,
  - but increase CPU
- offloaded compression

monitor:

get monitoring package
understand workload
baselining
find bottleneck
performance goal: low latency

web server: main function, retrieve resources off the system and deliver to the client. (efficiently and scale)

servers have dinite capacity, scale out(most) or scale up
tuning the web server:
- 1. have pre-allocated resources, Multi-processing Modules or MPM.
    - These are resources that are at the ready to respond to client requests, workers, and threads, and processes.
      - creating a new process in an operating system can be time consuming
      - do that work ahead of time and have the process ready, then clients don’t have to wait for that process to be created.
      - It’s already there for it to be consumed.
      - As connections come in they consume these pre-allocated workers
      - and tell Apache to keep creating new workers in the background, keep a fixed number of pre-allocated workers in line for those new clients to connect to.
    - different types of MPM modules: pre-allocate the whole process/connection or threads/connection.
    - All of this process pre-allocation, and also those working processes that are actually doing the work, require memory and CPU resources, so it’s up to us to ensure that we have a sufficient amount of capacity in our server hardware to support this configuration.
    - The out of the box configuration for MPM and Apache is sufficient for most workloads, but if you’re running a high capacity site this is usually one of the first stops that you’re going to want to look at in terms of performance tuning.
- 1. Another tuning fundamental: turning things off.
    - Turn off the things that you don’t need in your configuration, and the number one place to look is modules.
    - Disable those modules that you aren’t using. consumer resources and increase attack surface area

client side procy

cache content: put the resource closer to the client, client be serverd by the proxy Suid Proxy Sercer
- reduce respond time and reduce consumption of resource or network and web server. more

  
httpd -V
Server version: Apache/2.4.37 (centos)
Server built:   Dec 23 2019 20:45:34
Server's Module Magic Number: 20120211:83
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)


vi /etc/httpd/conf.modules.d/00-mpm.conf
# load module

ps --forest -ef | grep httpd
root      1056     1  0 12:13 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1117  1056  0 12:13 ?        00:00:00  \_ /usr/sbin/httpd -DFOREGROUND
apache    1121  1056  0 12:13 ?        00:00:00  \_ /usr/sbin/httpd -DFOREGROUND
apache    1122  1056  0 12:13 ?        00:00:00  \_ /usr/sbin/httpd -DFOREGROUND
apache    1123  1056  0 12:13 ?        00:00:00  \_ /usr/sbin/httpd -DFOREGROUND
root      3742  3191  0 12:54 pts/0    00:00:00                  \_ grep --color=auto httpd

# check the module using
httpd -D DUMP_MODULES
httpd -D DUMP_MODULES | wc -l

enable compression

  
$ vi /etc/httpd/conf.d/server1.demo.local.conf

<VirtualHost *:80>
    DocumentRoot "/var/www/html/server1.demo.local"
    ServerName server1.demo.local
    <IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/html
    </IfModule>
</VirtualHost>

httpd -D DUMP_VHOSTS
systemctl restart httpd

$ curl -I https://server1.demo.local/bigfile.html
HTTP/1.1 200 OK
Date: Sun, 24 May 2020 18:21:16 GMT
Server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
Last-Modified: Sun, 24 May 2020 17:53:42 GMT
ETag: "df6-5a6688b02390d"
Accept-Ranges: bytes
Content-Length: 3574
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8

$ curl -I https://server1.demo.local/bigfile.html --compressed
HTTP/1.1 200 OK
Date: Sun, 24 May 2020 18:21:33 GMT
Server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
Last-Modified: Sun, 24 May 2020 17:53:42 GMT
ETag: "df6-5a6688b02390d-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1238
Content-Type: text/html; charset=UTF-8

install and configure Squid Proxy Server

client - proxy - server

web content cache for client
dater response time
reduce bandwidth
security boundary
logs access
2 type:
- forward proxy:
  - client to squid
  - cache content o the client side
  - tcp 3128
- reverse proxy:
  - web server side
  - content cache for server, reply to web request and cache content
  - tcp 80

1. install and configure the Squid proxy server

  
server0:
yum install squid -y

# confirm hosts file
vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.1     server0.demo.local servr0
192.168.3.100   server1.demo.local server1
192.168.2.100   server2.demo.local server2
192.168.3.100   www.demo.local www


firewall-cmd --add-port=3128/tcp --permanent
firewall-cmd --reload

systemctl start squid
systemctl status squid


server1:
# curl will send http request over server0 3128 port for squid proxy
export http_proxy=https://server0.demo.local:3128
curl https://www.centinosystems.com
curl https://server0.demo.local:3128



server0: check log
more /var/log/squid/access.log
more /var/log/squid/cache.log



/etc/squid/
- squid.conf - main config file
- squid.conf.default

/etc/squid
cachemgr.conf
cachemgr.conf.default
errorpage.css
errorpage.css.default
mime.conf
mime.conf.default
squid.conf
squid.conf.default


/var.log/squid
- access.log
- cache.log
- squid -k rotate # rotate the logs
- time, duration, client IP, result code, bytes, request method, url, hierarchy code, type

2. restrict access to the Squid proxy server

access controls: in squid.conf.default

ACL elements:
- acl localnet src 192.168.0.0/16
access lists:
- http_access allow localnet

  
domain:
acl BackToWork dstdomain www.facebook.com
http_access deny BackToWork

Network:
acl NoInternet src 192.168.2.0/25
http_access deny NoInternet

process in order

  
1. to stop domain

server0:
vi /etc/squid/squid.conf   # all acl rule
# add line
acl BackToWork dstdomain www.facebook.com
# under insert section:
http_access deny BackToWork

systemctl restart squid


server1:
# check hosts file
export http_proxy=https://server0.demo.local:3128
curl https://facebook.com


2. to stop all network of server2

server0:
# check log
more /var/log/squid/access.log
more /var/log/squid/cache.log

# to stop all network of server2
vi /etc/squid/squid.conf   # all acl rule
# add line
acl intnet2 src 192.168.2.0/25
# put before localnet
http_access deny intnet2  # deny access from ...

systemctl restart squid

curl https://google.com

cache
- caching to disk in not enables bu default, only to memory.
  - cache_dir ufd /var/to/cache SizeMB L1 L2
  - cache_dir ufd /var/spool/squid 100 16 256
- specify more than one cache_dir
- likely on different disks
- owned by squid:squid
- SELinux content
  - system_u:object_r:squid_cache_t:s0

  
1. add cache dir

sudo su -
vi /etc/squid/squid.conf
# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/spool/squid 100 16 256

systemctl restart squid

ls -la /var/spool/
drwxr-xr-x. 11 root  root   122 May 24 14:37 .
drwxr-xr-x. 21 root  root  4096 May 20 13:18 ..
drwxr-x---. 18 squid squid  184 May 24 17:24 squid

# check SELinux file content -Z
ls -laZ /var/spool/
drwxr-xr-x. 11 root  root  system_u:object_r:var_spool_t:s0          122 May 24 14:37 .
drwxr-xr-x. 21 root  root  system_u:object_r:var_t:s0               4096 May 20 13:18 ..
drwxr-x---. 18 squid squid system_u:object_r:squid_cache_t:s0        184 May 24 17:24 squid



2. move web cache to another dir

mkdir /var/spool/cache1
chown squid:squid /var/spool/cache1/
chmod 750 /var/spool/cache1/

semanage fcontext -at squid_cache_t -s system_u "/var/spool/cache1(/.*)?"
restorecon -RFv /var/spool/cache1

ls -laZ /var/spool/
drwxr-xr-x. 12 root  root  system_u:object_r:var_spool_t:s0          136 May 24 17:26 .
drwxr-xr-x. 21 root  root  system_u:object_r:var_t:s0               4096 May 20 13:18 ..
drwxr-x---.  2 squid squid system_u:object_r:squid_cache_t:s0          6 May 24 17:26 cache1
drwxr-x---. 18 squid squid system_u:object_r:squid_cache_t:s0        184 May 24 17:24 squid

vi /etc/squid/squid.conf
cache_dir ufs /var/spool/cache1 100 16 256

systemctl restart squid

ls -la /var/spool/cache1/
ls /var/spool/cache1/
00  01  02  03  04  05  06  07  08  09  0A  0B  0C  0D  0E  0F  swap.state

configure http client to automatically use proxy server

transparent proxy / intercept proxy

configure each client, so many work
use network to redirect traffic to the proxy
common implement on a network gateway
ipatbles/firewalld to REDIRECT output HTTP requests (outbound trafiic) to the proxy on 3128 (squid)

  
server0:

vi /etc/squid/squid.conf
# add port
# Squid normally listens to port 3128
http_port 3128
http_port 3129 intercept

firewall-cmd --add-forward-port=port=80:proto=tcp:toport=3129:toaddr=192.168.1.1
# tcp 80 -> 3129 192.168.1.1
firewall-cmd --add-port=3129/tcp

netstat -plant



server1:
# without config
# export http_proxy=https://server0.demo.local:3128
env | grep http_proxy

curl https://www.centinosystems.com


server0:
tail /var/log/squid/access.log

apache modules

module:

add functionality to server
many module available
load only needed, more module, more CPU memory resource used.
write your own module

get the module: yum / package manager
- yum install mod_ssl
direct download

location: /etc/httpd/modules -> /usr/lib64/httpd/modules config file: /etc/httpd/conf.modules.d/ configuration: LoadModule ssl_mode modules/mod_ssl.so

common used modules:

  
ls -lah
# basic configure for apache
drwxr-xr-x.   2 root root   37 May 24 18:24 conf
# runtime
drwxr-xr-x.   2 root root  100 May 24 18:24 conf.d
drwxr-xr-x.   2 root root  226 May 24 18:24 conf.modules.d
lrwxrwxrwx.   1 root root   19 Dec 23 15:46 logs -> ../../var/log/httpd
lrwxrwxrwx.   1 root root   29 Dec 23 15:46 modules -> ../../usr/lib64/httpd/modules
lrwxrwxrwx.   1 root root   10 Dec 23 15:46 run -> /run/httpd
lrwxrwxrwx.   1 root root   19 Dec 23 15:46 state -> ../../var/lib/httpd


cat conf/httpd.conf
# Include conf.modules.d/*.conf


ls -la ./conf.modules.d/
drwxr-xr-x. 2 root root  226 May 24 18:24 .
drwxr-xr-x. 5 root root  105 May 24 18:24 ..
-rw-r--r--. 1 root root 3311 Dec 23 15:44 00-base.conf  # list of module
-rw-r--r--. 1 root root  139 Dec 23 15:44 00-dav.conf
-rw-r--r--. 1 root root   41 Dec 23 15:44 00-lua.conf
-rw-r--r--. 1 root root  948 Dec 23 15:44 00-mpm.conf
-rw-r--r--. 1 root root  787 Dec 23 15:44 00-optional.conf
-rw-r--r--. 1 root root 1073 Dec 23 15:44 00-proxy.conf
-rw-r--r--. 1 root root   88 Dec 23 15:44 00-systemd.conf
-rw-r--r--. 1 root root  451 Dec 23 15:44 01-cgi.conf
-rw-r--r--. 1 root root   45 Nov 15  2019 10-h2.conf
-rw-r--r--. 1 root root   57 Nov 15  2019 10-proxy_h2.conf
-rw-r--r--. 1 root root  496 Dec 23 15:46 README


httpd -D DUMP_MODULES | more
httpd -M

yum install mod_ssl

$ rpmquery -ql mod_ssl
/etc/httpd/conf.d/ssl.conf
/etc/httpd/conf.modules.d/00-ssl.conf # loadmodule
/usr/lib/.build-id
/usr/lib/.build-id/05/263a2bbd373d4458d022df54a3da110453a85a
/usr/lib/systemd/system/httpd-init.service
/usr/lib/systemd/system/httpd.socket.d/10-listen443.conf
/usr/lib64/httpd/modules/mod_ssl.so  # phtsical path
/usr/libexec/httpd-ssl-gencerts
/usr/libexec/httpd-ssl-pass-dialog
/usr/share/man/man8/httpd-init.service.8.gz
/var/cache/httpd/ssl

$ more /etc/httpd/conf.modules.d/00-ssl.conf
LoadModule ssl_module modules/mod_ssl.so

more /etc/httpd/conf.d/ssl.conf

mod_alias

update
http to https
new domain

  
# make sure installed
grep mod_alias /etc/httpd/conf.modules.d/*
/etc/httpd/conf.modules.d/00-base.conf:LoadModule alias_module modules/mod_alias.so

# check loaded
httpd -D DUMP_MODULES | grep alias
 alias_module (shared)
 vhost_alias_module (shared)

# module config in virtual hosts config
cd /etc/httpd/conf.d
vi server1.demo.local.conf
# add module
<VirtualHost *:80>
    DocumentRoot "/var/www/html/server1.demo.local"
    ServerName server1.demo.local
    <IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/html
    </IfModule>
    <IfModule alias_module>
        Redirect permanent /products /newproducts
    </IfModule>
</VirtualHost>

systemctl restart httpd


mkdir /var/www/html/server1.demo.local/products
mkdir /var/www/html/server1.demo.local/newproducts

echo products > /var/www/html/server1.demo.local/products/index.html
echo newproducts > /var/www/html/server1.demo.local/newproducts/index.html


curl https://server1.demo.local/products
# willnot redirect in terminal, but browser will
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://server1.demo.local/newproducts">here</a>.</p>
</body></html>

cgi application

CGI

static vs dynamic content
execute code on the server side
common gateway interface
content is returned to the client

  
LoadModule cgi_module modules/mod_cgi.so
# separate the documentroot and application
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

cgi sceipt

  
#!/bin/bash
echo "Content-type: text/html"
echo ""
echo "The hostname of this server is: `hostname` on `date`"

  
# check loaded
httpd -D DUMP_MODULES | grep cgi

# check the dir for cgi
vi /etc/httpd/conf/httpd.conf
# check
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"


getsebool httpd_enable_cgi
httpd_enable_cgi --> on


cd /var/www/cgi-bin/
vi hostname.sh
chmod +x hostname.sh

# allow to run the script
curl https://server1.demo.local/cgi-bin/hostname.sh
# output
The hostname of this server is: server1.demo.local on Sun May 24 18:55:35 EDT 2020

configure proxying and caching for HTTP Services

2 type:
- forward proxy:
  - client to squid
  - cache content o the client side
  - tcp 3128
- reverse proxy:
  - web server side
  - content cache for server, reply to web request and cache content
  - tcp 80

squid, varnish, nginx…

using squid as a reverse proxy

web acceleration
basic load balancing
SSL termination: move ssl and encrypt to proxy
domain URL routing
…

  
server0:

// config host entry
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.1     server0.demo.local server0
192.168.3.100   server1.demo.local server1
192.168.2.100   server2.demo.local server2
192.168.1.1     proxy.demo.local


vi /etc/squid/squid.conf
// add port
http_port 80 accel defaultsite=no-vhost
cache_peer server1.demo.local parent 80 0 no-query \ originserver name=PSReverseProxy
// INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl demo_site dstdomain proxy.demo.local
http_access allow demo_site

cache_peer_access PSReverseProxy allow demo_site
cache_peer_access PSReverseProxy deny all


netstat -plant

firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --reload


// go to 192.168.1.1, proxy,
// receive by reverse proxy on port 80
// reverse proxy reachout to server1 ask for content
// and send it back

curl https://proxy.demo.local

00CodeNote, LinuxNote

This post is licensed under CC BY 4.0 by the author.