Knowledgebase

In the world of modern web hosting and VPS management, having an effective logging and monitoring system is essential for tracking system performance, diagnosing issues, and ensuring overall security. The ELK Stack (Elasticsearch, Logstash, and Kibana) is a powerful open-source toolset that allows you to collect, process, and visualize logs in real time.

In this guide, we will walk you through the steps to set up the ELK Stack on your AnonVM VPS. Once the system is set up, you'll be able to monitor server logs, visualize traffic, and gain insights into server performance.

Prerequisites

Before you begin, ensure you have:

• AnonVM VPS running a compatible Linux distribution (Ubuntu 20.04 or later recommended).
• Root access or sudo privileges.
• Java installed on your VPS (since Elasticsearch and Logstash require it).
• A domain name (optional) to make accessing Kibana easier.
• Basic knowledge of the command line.

Step 1: Install Java

The ELK Stack requires Java to run, particularly for Elasticsearch and Logstash.

To install Java, run the following commands:

Make sure Java 11 or later is installed.

Step 2: Install Elasticsearch

Elasticsearch is the core component of the ELK Stack that stores and indexes logs and metrics. Follow these steps to install it:

1. Add the Elasticsearch GPG Key:

    

   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2. Add the Elasticsearch repository:

    

   sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'

3. Install Elasticsearch:

    

   sudo apt update sudo apt install elasticsearch -y

4. Enable and start Elasticsearch:

    

   sudo systemctl enable elasticsearch sudo systemctl start elasticsearch

5. Test Elasticsearch: Once Elasticsearch is running, you can test it by sending a request to the API:

    

   curl -X GET "localhost:9200/"

   You should see a JSON response containing information about your Elasticsearch node.

Step 3: Install Logstash

Logstash is responsible for collecting, parsing, and transforming logs before sending them to Elasticsearch.

1. Install Logstash:

    

   sudo apt install logstash -y

2. Configure Logstash to Collect Logs: Create a configuration file to define how Logstash processes logs.

   Example:

    

   sudo nano /etc/logstash/conf.d/logstash.conf

   Add the following configuration to collect system logs (you can adjust paths as needed):

    

   input { file { path => "/var/log/syslog" start_position => "beginning" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "syslog-%{+YYYY.MM.dd}" } }

3. Start Logstash:

    

   sudo systemctl start logstash

4. Enable Logstash to start on boot:

    

   sudo systemctl enable logstash

Step 4: Install Kibana

Kibana provides a user-friendly interface for visualizing and analyzing your logs stored in Elasticsearch.

1. Install Kibana:

    

   sudo apt install kibana -y

2. Start Kibana:

    

   sudo systemctl start kibana

3. Enable Kibana to start on boot:

    

   sudo systemctl enable kibana

4. Access Kibana: By default, Kibana runs on port 5601. You can access it via your web browser by navigating to:

   arduino

    

   http://<your_vps_ip>:5601

   If you have a domain set up, use your domain name:

   arduino

    

   http://yourdomain.com:5601

Step 5: Configure Kibana

1. Log in to Kibana: Use your browser to go to http://<your_vps_ip>:5601 or http://yourdomain.com:5601.

2. Set the Default Index Pattern:

   • Navigate to Management > Kibana > Index Patterns.
   • Create a new index pattern. For example, enter syslog-* if you are collecting logs from syslog.
   • Select the timestamp field (@timestamp by default) and click Create index pattern.

Step 6: Configure Firewall (Optional but Recommended)

To secure your Kibana interface, it is advisable to allow only trusted IPs to access it. If you're using UFW (Uncomplicated Firewall), you can configure the firewall as follows:

1. Allow HTTP (80) and HTTPS (443) ports:

    

   sudo ufw allow 80,443/tcp

2. Allow Kibana port (5601) only for trusted IPs:

    

   sudo ufw allow from <trusted_ip> to any port 5601

3. Enable UFW:

    

   sudo ufw enable

Step 7: View Logs in Kibana

Once your system is configured and logs are flowing into Elasticsearch, you can view them in Kibana.

1. Go to the Discover page in Kibana.
2. You should now see your log data (e.g., syslog-*).
3. Use the search bar to query logs and visualize data.
   • For example, type error to view any errors in your system logs.

Step 8: Set Up Logstash Inputs for Additional Logs

You can configure Logstash to collect additional logs (such as Nginx or Apache logs) by adding more input files.

1. Create a new Logstash configuration file:

    

   sudo nano /etc/logstash/conf.d/nginx.conf

2. Add the configuration to collect Nginx logs:

    

   input { file { path => "/var/log/nginx/access.log" start_position => "beginning" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "nginx-%{+YYYY.MM.dd}" } }

3. Restart Logstash:

    

   sudo systemctl restart logstash

Step 9: Set Up Alerts and Dashboards (Optional)

Once your logs are collected in Elasticsearch, you can create custom Kibana dashboards and set up alerts to monitor system performance, errors, and other important metrics.

1. Create Dashboards: Kibana allows you to create visualizations and dashboards to monitor specific metrics, such as CPU usage, network traffic, or application errors.

2. Set Alerts: Kibana can send alerts when certain thresholds are met (e.g., a high error rate or resource usage). Use the Alerting feature under the Management section to set these up.

Conclusion

By setting up the ELK Stack on your AnonVM VPS, you’ve gained powerful logging and monitoring capabilities. Elasticsearch indexes your logs, Logstash processes them, and Kibana allows you to visualize and analyze them in real time. With this setup, you can keep an eye on system performance, troubleshoot issues efficiently, and even set up alerts to stay ahead of potential problems.

The ELK Stack provides the flexibility to scale and customize it for various use cases, from security monitoring to performance analytics. It’s an essential toolset for any server administrator or developer who values insights into system health and security.