In real production environments, developers and testers strive to ensure security to prevent critical information from leaking or being stolen. Many enterprises or organizations have more comprehensive threat modeling, yet security issues still persist incessantly. Some companies expect to rely on white hats or black hats on the internet to help them build their corporate security. However, companies need to understand that not all white hats or black hats are willing to share their security advice with them. Security measures are not as simple as using 2FA and password managers, installing updates, and avoiding suspicious links.
The content of this article has been presented by me twice at some security conferences. Here, I have made some modifications to parts of the content and wish to share them once again, primarily for those who are new to learning or understanding Linux security. In this article, I will not discuss web application security. For those interested in learning this area, you can refer to the OWASP Top 10, which teaches developers and testers about SQL injection, CSRF, XSS, session management, and more. If you are concerned about more advanced attackers and need to guard against the possible execution of malicious code or privilege escalation scripts, then you will learn what you need to know here. Remote intruders, or even a trivial insider threat, may have already lurked within your system waiting for an opportunity.
Below, I will mention some basic concepts and include related tools. Meanwhile, you need to ask yourself:
ACL (Access Control List)
Is your infrastructure divided into different systems with different role-based access levels, or do users have the exact same privileges?
acl: getfacl+setfacl
System Audit
Have you thoughtfully considered which users are logging into a machine?
Check: auditd, go-audit
Static Analysis and Fuzzing
You may be running C/C++. Memory safety issues occupy most of the vulnerabilities that arise. Have you audited, statically analyzed, or fuzzed these codes to ensure their security? Have you considered developing in Rust?
Check: afl, Radamsa, Sulley, boofuzz, Coverity Scan, Valgrind, sanitizers
Network Segmentation
Have you properly separated your backend from your frontend and load balancers? Have you taken the time to set up a corporate VPN and assign your machines private internal addresses?
Check: iptables, ufw
Isolation
Do your employees use their work computers for personal activities such as gaming or unrelated applications? Or do they use something like virtual machines or containers for messaging, browsing, developing�
Check: Qubes, VirtualBox
File Permissions and umask
Anyone with a basic understanding of Unix should be familiar with file permissions. If you do not strictly control the read, write, and execute permissions of certain files, I strongly suggest you implement permission controls according to the strictest specifications.
More content
Containers
Containerization theoretically aids security greatly. But I wonder who is allowed to build and deploy images into production environments, and whether they have also signed, verified, and monitored security updates and CVEs?
Check: Docker, LXC
Leveraging Threat Intelligence
Malicious IP flows originating from the IPv4 address space, botnets preparing for DDoS attacks, and automated exploit scans are ongoing. You can collect intelligence about such activities yourself or subscribe to feeds or blacklist lists of related products. But do your endpoints/firewalls react to and integrate this information?
Check: Threat intelligence resource lists
Firewall and Packet Filtering
How often do you review your iptables rules, or do you understand what your router/firewall is implementing, and have you tested their operation? Have you correctly configured packet filtering rules to ensure that machines only communicate with necessary data?
Check: pfSense, OPNsense
DNS and Domain Registrars
How much effort have you put into locking this down? How much effort have you spent securing it? Are you receiving alerts when your domain server or DNS zone files are altered without authorization? Furthermore, have you enabled DNSSEC?
Personally, I recommend using Namecheap as a registrar and Cloudflare for high-performance DNS.
Physical Access
If I were a law enforcement officer, data center staff, or your hosting provider, would I freely read the contents of your server (except for full disk encryption)? When your disks retire or are replaced, are the contents erased? Would you receive an alert if someone plugged a USB into the 1U rack server?
Check: LUKS/cryptsetup
Do You Have Deterministic Builds?
When developers push code builds to production, can you verify the real intent of those codes and ensure that the source code or statically linked dependencies have not been maliciously modified?
Check: Gitian
Verify Digital Signatures
Undoubtedly, you may obtain or download some code and software from unofficial sites. Before continuing to build or install, have you compared checksums/hashes or verified the signatures of the download?
Check: Creating and verifying signatures with GnuPG
Sandbox Environment
Does it have AppArmor profiles, seccomp filters, or RBAC policies specifying its capabilities in terms of system calls and access permissions?
Check: seccomp, AppArmor
TLS and Encryption Configuration
Have you completely decommissioned those insecure cipher suites and algorithms (like MD5, SHA1, RC4) and chosen the best encryption method, HMAC, and key exchange algorithms based on compatibility and user expectations? If available, prioritize using RSA encryption algorithms. This applies to OpenSSH, GnuPG, OpenVPN, etc. With Letâs Encrypt, you can easily obtain free SSL certificates.
Check: Apply password hardening, SSL and TLS, server-side TLS
Public and Secret Key Management
If every employee has their key, consider synchronizing them throughout the domain and moving keys out of version control.
Check: GPGSync, sops, Vault
HTTP Header Security
Regarding HTTP header security, here is a list for reference: X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, X-Download-Options, X-Permitted-Cross-Domain-Policies, Content-Security-Policy, Referrer-Policy, Strict-Transport-Security, Public-Key-Pins. Correct configuration of these response headers is also crucial for security.
Check: securityheaders.io, Mozilla web security guidelines
File Integrity Monitoring
Do you periodically check important files to ensure they havenât been modified and generate alerts when changes are detected?
Check: Tripwire, OSSEC
Intrusion Detection
Perhaps you have deployed various types of intrusion detection tools, but are you just running them with default rule configurations without taking the time to learn the technical knowledge and configure a suitable rule set based on your application scenario?
Check: Compare host-based intrusion detection systems, Snort
Vulnerability Management
By subscribing to emails, we acquire new vulnerability reports and fix vulnerabilities. So, do you recall the last time you reviewed CVE activity?
Check: Nessus, CoreOS clair
Securing Fundamental Systems
Do you truly trust that Debian/Ubuntu, RHEL or any companyâs third-party software repositories can consistently provide you with non-malicious packages? Hereâs an idea: You can host your repository, stick to specific versions, and only upgrade after testing.
Or, you can run a minimal operating system based on Alpine or LinuxKit, minimizing your attack surface to the maximum extent.
LSM (Linux Security Modules)
Such as AppArmor or SELINUX, have they recently done anything for you?
Linux Kernel Hardening
Here I suggest everyone to directly check out PaX and grsecurity patches. The recent Spectre and Meltdown vulnerabilities have also sounded an alarm for our kernel security issues.
Check: grsecurity.net, Linux Kernel Security Hardener, Kernel Self Protection Project
Remove Unnecessary Devices
If you are not using Thunderbolt, Firewire, wireless cards, or any modules with DMA (Direct Memory Access) mode, then you have no reason to load these kernel modules.
Check: Kernel module blacklist
Log Filtering
Perhaps you are still checking log files manually, then you should learn to write Logstash filters and grok patterns. Grok is the most critical plugin in Logstash. You can predefine named regular expressions in grok and reference them later (grok parameters or other regular expressions). It is highly suitable for syslog logs, apache, and some other webserver logs, as well as mysql logs. Grok offers many pre-defined patterns, and you can also define your own.
Check: Filebeat, rsyslog, Logstash
How Do You Monitor Resource Usage?
RAM, CPU load, available disk space. Although these seem trivial, these areas are crucial for detecting abnormal activities and thus worth mentioning.
Check: Metricbeat, Prometheus node_exporter, Nagios, Osquery
Infrastructure Testing
People are familiar with various aspects of software testing but not with infrastructure testing. How do you ensure that your system state always meets your expectations?
Check: Serverspec, Testinfra
Platform and Firmware Security
Your BIOS and other low-level interfaces are subject to errors. IntelÂź AMT and Management Engine, as well as Computrace should be disabled. The link below provides a very useful framework for analyzing system firmware and hardware component security.
Check: CHIPSEC
Securing Remote Shell
General guidelines for sshd: Disable root login, use keys instead of passwords, and set brute force protection. A better solution is to place it behind a VPN, through a validated Tor hidden service, or require port knocking to dynamically open specific port connections to the host.
Check: fail2ban, denyhosts, sshguard, Secure Secure Shell
Webserver Best Practices
If you donât want to disclose the version information of the current server you are running, for nginx you can achieve this by setting the server_tokens parameter to off; for Apache, setting the ServerSignature to off is sufficient.
When running complex applications that rely on dynamic scripting languages, consider running a WAF (Web Application Firewall) like ModSecurity. Cloudflare provides this service extensively to its clients.
Secondary Factor
I strongly recommend using YubiKey; it has many useful features. It can be configured to output static passwords (ideal for PAM user logins or mounting volume encryption), HOTP, or universal two-factor (U2F) authentication, or used as an OpenPGP smart card. These devices are indispensable for any system administrator. To this end, I have already published detailed installation guides for YubiKey GPG+SSH.
DNS Resolution
Is your /etc/resolv.conf file correctly configured? Quad9 is an alternative product to Googleâs Public DNS or OpenDNS, blocking client access to malicious domain names, similar to how Chromeâs Safe Browsing feature protects users. Hence, setting your domain server to 9.9.9.9 will better safeguard your security.
Audit Trusted Parties
In addition to keeping the root certificate store of the system updated, you should also periodically check the package manager to see which third parties are trusted, whether their repository signing keys are strong enough (many still use 1024-bit DSA), and remove those that are expired.
Try: apt-key list, rpm -qa gpg-pubkey
Sign Git Commits and Tags
Currently, almost everyone uses git for version control. When you publish new versions, are they based on GPG signed git tags? If you like, you can also sign commits.
Check: Signing tags with GPG, Git signing, Git toolsâââSigning Your Work