Welcome to CrazyWall

Think of CrazyWall as a digital evidence board. It accepts and treats different user inputs, namely IP address, username, full name, domain name. after detecting and processing your input based on your input, it returns different data.
The complete writeup is available here.

Demo

How to use it?

All you have to do is go to the main page, type your keyword, select your mode between databases and live and hit the enter button. Valid inputs are : a domain name, e.g., domain.com a phone number (international format), e.g., 00123456789 a full name, e.g., john due a user name, e.g., sherlock an IP address, e.g., 1.2.3.4 a keyword for live search, e.g., crypto. Please watch the demo for complete usage.

ScreenShots

Search

Live News

Live Social

UserName Data

Site Data

Motivation

The primary motivation was to create a PoC toward incredible real-time threat intel; the original plan was to re-write this (cut out the external APIs and add their functionality to the back-end as an engine) embed it into our Suite. Then decided to release a chunk of research as a standalone suite in front of you called CrazyWall.

Feature

Hand-picked sources: data sources are synergized
Cross-data-search: getting all associations, like addresses, e-mails, phone numbers, etc., with a particular entity.
Stand alone API: gathers all external resources under a unified, simple-to-use API to feed/pipe them to your existing toolkit or machine learning models without the need to use the front end.

Data Sources

Version

1.0.0

Development

in this section, you can learn how this tool is developed and deployed.

Front-End

The front-end is developed using Vue.js 2.x / Nuxt.js; all you have to do is make sure you install Node.js 14+ on your target machine.
Go to the front-end directory. It uses client-side rendering and static site generation as it requires no SEO and brings complete separation from the back-end.

Using yarn:

Dev:

yarn
yarn dev

Build:

yarn build 
yarn start

using npm

Dev:

npm i 
npm run dev

Build:

npm run build 
npm run start

Back-End

The back-end is developed in Golang, using Fiber and Gorm and MySQl8. To work on the back-end, you have to install golang 1.16+ to get the same hot-reloading as the front-end while developing; you can do the following. To start back-end, you need to copy the .env.example file to .env and then obtain the required API. A majority of them offer a free API. check out data source section for more information.

Using fiber CLI

fiber dev

The deployment

you can deploy it using managed or manual. Please note that this project is meant to be a PoC, so you are responsible for thoroughly reviewing and testing the code before use in any non-testing environment.

Managed

For managed services, which are more painless, you can use Heroku, Google App Engine, AWS Elastic Beanstalk, DigitalOcean App Platform. They all have app detection, and you can follow their guides on how to deploy Golang, NodeJS, and MySql8.

Manual

Here is an example of manual deployment on Ubuntu 20.04. install Nginx, Node 14+ and Golang 13+ using apt.

Nginx

Here is an example of the Nginx configuration with let’s encrypt (certbot) SSL, which handles both front-end and API.

map $sent_http_content_type $expires {
    "text/html"                 epoch;
    "text/html; charset=utf-8"  epoch;
    default                     off;
}

server {             # the port Nginx is listening on
    server_name     crazywall.io www.crazywall.io;    # setup your domain here

    gzip            on;
    gzip_types      text/plain application/xml text/css application/javascript;
    gzip_min_length 1000;

    location / {
        expires $expires;

        proxy_redirect                      off;
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_read_timeout          1m;
        proxy_connect_timeout       1m;
        proxy_pass                          http://127.0.0.1:3000;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/crazywall.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/crazywall.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


}

server {
    server_name api.crazywall.io;

    location / {
        proxy_pass http://localhost:3333;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/crazywall.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/crazywall.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

Front-End persitance service

touch /etc/systemd/system/front.service

[Unit]
After=network.service
Description="Front End"
[Service]
# current working directory (where your app lives)
WorkingDirectory=/root/app/crazymap-frontend

# setting ENV variables
Environment=NODE_ENV=production BASE_URL=https://api.crazywall.io

# start your app with the command
ExecStart=yarn start

# always restart the app if it crashes
Restart=always

# if the crash happens, wait 3 seconds before restarting
RestartSec=3

# Run this service automatically after boot:
[Install]
WantedBy=multi-user.target

Back-end persistent service

touch /etc/systemd/system/api.service

[Unit]
Description=CrazyWall Api

[Service]
Type=simple
Restart=always
RestartSec=5s
WorkingDirectory=/root/app/crazymap-backend
ExecStart=/root/app/crazymap-backend/api
Environment=DATABASE_HOST="127.0.0.1"
Environment=DATABASE_PORT=3306


[Install]
WantedBy=multi-user.target

finally enable them and reload

systemctl daemon-reload
nginx -s reload
systemctl enable api
systemctl enable front

GitHub

View Github