Shuffle

Shuffle is an Open Source SOAR I really appreciate as it does focus on CyberSecurity. To know more about it go there: https://medium.com/shuffle-automation Or there : Official Website

Hi Frikky !!! Hopefully, you will appreciate what you are seing here ^^

Preparaton

Setup data locations

Setup Docker Swarm

Deploying Portainer and the Portainer Agent to manage a Swarm cluster is easy! You can directly deploy Portainer as a service in your Docker cluster. Note that this method will automatically deploy a single instance of the Portainer Server, and deploy the Portainer Agent as a global service on every node in your cluster.

# mkdir -p /mnt/configuration_data/compose_files/SOAR

Customisation

At the time of this writing I may be the only one shufflying around with swarm :p So there is not yet a stack ready compose file.

Let's create one /mnt/configuration_data/compose_files/SOAR/shuffle.stack.yml

version: '3.8'
services:
  backend:
    image: ghcr.io/frikky/shuffle-backend:latest
    environment:
      BACKEND_HOSTNAME: backend
      BACKEND_PORT: '5001'
      DATASTORE_EMULATOR_HOST: shuffle-database:8000
      DB_LOCATION: ./shuffle-database
      ENVIRONMENT_NAME: Shuffle
      FRONTEND_PORT: '3001'
      FRONTEND_PORT_HTTPS: '3443'
      HTTPS_PROXY: ''
      HTTP_PROXY: ''
      ORG_ID: Shuffle
      OUTER_HOSTNAME: backend
      SHUFFLE_APP_DOWNLOAD_LOCATION: https://github.com/frikky/shuffle-apps
      SHUFFLE_APP_FORCE_UPDATE: 'false'
      SHUFFLE_APP_HOTLOAD_FOLDER: /shuffle-apps
      SHUFFLE_APP_HOTLOAD_LOCATION: ./shuffle-apps
      SHUFFLE_BASE_IMAGE_NAME: frikky
      SHUFFLE_BASE_IMAGE_REGISTRY: ghcr.io
      SHUFFLE_BASE_IMAGE_TAG_SUFFIX: '-0.9.30'
      SHUFFLE_CONTAINER_AUTO_CLEANUP: 'true'
      SHUFFLE_DEFAULT_APIKEY: ''
      SHUFFLE_DOWNLOAD_AUTH_BRANCH: ''
      SHUFFLE_DOWNLOAD_AUTH_PASSWORD: ''
      SHUFFLE_DOWNLOAD_AUTH_USERNAME: ''
      SHUFFLE_DOWNLOAD_WORKFLOW_BRANCH: ''
      SHUFFLE_DOWNLOAD_WORKFLOW_LOCATION: ''
      SHUFFLE_DOWNLOAD_WORKFLOW_PASSWORD: ''
      SHUFFLE_DOWNLOAD_WORKFLOW_USERNAME: ''
      SHUFFLE_ELASTIC: 'true'
      SHUFFLE_FILE_LOCATION: /shuffle-files
      SHUFFLE_OPENSEARCH_APIKEY: ''
      SHUFFLE_OPENSEARCH_CERTIFICATE_FILE: ''
      SHUFFLE_OPENSEARCH_CLOUDID: ''
      SHUFFLE_OPENSEARCH_PROXY: ''
      SHUFFLE_OPENSEARCH_SKIPSSL_VERIFY: 'true'
      SHUFFLE_OPENSEARCH_URL: http://opensearch:9200
      SHUFFLE_PASS_APP_PROXY: 'FALSE'
      SHUFFLE_PASS_WORKER_PROXY: 'TRUE'
      SHUFFLE_ENCRYPTION_MODIFIER: a7cee5f9-8e55-4e05-98d4-59aa9595339b
      BASE_URL: https://shuffle.lab.local
      SSO_REDIRECT_URL: https://shuffle.lab.local
    #ports:
    # - 5001:5001
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /mnt/swarm/container_data/Shuffle/shuffle-apps:/shuffle-apps
     - /mnt/swarm/container_data/Shuffle/shuffle-files:/shuffle-files
    networks:
     - shuffle
    depends_on:
     - opensearch 
    logging:
      driver: json-file
    deploy:
      placement:
        constraints:
          - node.labels.localisation==PROD  
      resources:
        limits:
          #cpus: '0.5'
          memory: 8G        
  frontend:
    image: ghcr.io/frikky/shuffle-frontend:latest
    healthcheck:
      test: curl -fs http://localhost:80 || exit 1
      interval: 30s
      timeout: 5s
      retries: 3    
    environment:
      BACKEND_HOSTNAME: backend
      BASE_URL: https://shuffle.lab.local
      SSO_REDIRECT_URL: https://shuffle.lab.local
    #ports:
    # - 3001:80
    # - 3443:443
    networks:
     - shuffle
     - reverseproxy
    logging:
      driver: json-file
    depends_on:
      - backend
    deploy: 
      placement:
        constraints:
          - node.labels.localisation==PROD         
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.shuffle.rule=Host(`shuffle.lab.local`)"
        - "traefik.http.routers.shuffle.entrypoints=web-secure,web"
        - "traefik.http.routers.shuffle.tls=true"
        - "traefik.http.services.shuffle.loadbalancer.server.port=80"
        - "traefik.docker.network=reverseproxy"

  opensearch:
    image: opensearchproject/opensearch:2
    healthcheck:
      test: curl -fs http://localhost:9200/_cat/health || exit 1
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 45s    
    environment:
      OPENSEARCH_JAVA_OPTS: -Xms4096m -Xmx4096m
      bootstrap.memory_lock: 'true'
      cluster.initial_master_nodes: opensearch
      node.store.allow_mmap: 'false'
      cluster.name: shuffle-cluster
      cluster.routing.allocation.disk.threshold_enabled: 'false'
      discovery.seed_hosts: opensearch
      node.name: opensearch
      plugins.security.disabled: 'true'
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 
        hard: 65536      
    #ports:
     #- 9200:9200
    volumes:
     - /mnt/swarm/container_data/Shuffle/shuffle-database:/usr/share/opensearch/data
     #- /var/glusterfs/shuffle-database:/usr/share/opensearch/data
     - /mnt/swarm/container_data/Shuffle/shuffle-backup/opensearch:/mnt/backup
    networks:
     - shuffle
    logging:
      driver: json-file
    deploy:
      #endpoint_mode: dnsrr     
      placement:
        constraints:
          - node.labels.localisation==PROD 
      resources:
        limits:
          #cpus: '0.5'
          memory: 8G  

  orborus:
    #image: ghcr.io/frikky/shuffle-orborus:latest
    image: ghcr.io/frikky/shuffle-orborus:nightly
    environment:
      #SHUFFLE_APP_SDK_VERSION: 0.8.97
      SHUFFLE_WORKER_VERSION: latest     
      BASE_URL: http://tasks.backend:5001
      #BASE_URL: https://shuffle.lab.local
      CLEANUP: 'true'
      DOCKER_API_VERSION: '1.41'
      ENVIRONMENT_NAME: Shuffle
      HTTPS_PROXY: ''
      HTTP_PROXY: ''
      ORG_ID: Shuffle
      SHUFFLE_BASE_IMAGE_NAME: frikky
      SHUFFLE_BASE_IMAGE_REGISTRY: ghcr.io
      SHUFFLE_BASE_IMAGE_TAG_SUFFIX: -0.8.80
      SHUFFLE_ORBORUS_EXECUTION_CONCURRENCY: '50'
      SHUFFLE_ORBORUS_EXECUTION_TIMEOUT: '800'
      SHUFFLE_PASS_APP_PROXY: 'FALSE'
      SHUFFLE_PASS_WORKER_PROXY: 'FALSE'
      SHUFFLE_SCALE_REPLICAS: 2
      SHUFFLE_SWARM_CONFIG: run
      SHUFFLE_LOGS_DISABLED: "true"
      SHUFFLE_SWARM_NETWORK_NAME: shuffle
      SSO_REDIRECT_URL: https://shuffle.lab.local
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    networks:
     - shuffle
    depends_on:
      - backend     
    logging:
      driver: json-file
    deploy:
      placement:
        constraints:
          - node.labels.localisation==PROD

  oauth2-proxy:
     container_name: oauth2-proxy
     image: quay.io/oauth2-proxy/oauth2-proxy:latest
     command:  --config /oauth2-proxy.cfg
     hostname: oauth2-proxy
     networks:
     - shuffle
     - reverseproxy
     volumes:
       - /mnt/swarm/container_data/Shuffle/oauth2-proxy/oauth2-proxy.cfg:/oauth2-proxy.cfg
       - /mnt/swarm/container_data/Shuffle/oauth2-proxy/logs:/logs
       - /etc/ssl/certs/:/etc/ssl/certs/:ro
     restart: unless-stopped
     deploy: 
      placement:
        constraints:
          - node.labels.localisation==PROD         
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.opensearch-secure.rule=Host(`shuffle.lab.local`)"
        - "traefik.http.routers.opensearch-secure.entrypoints=opensearch-secure"
        - "traefik.http.routers.opensearch-secure.tls=true"
        - "traefik.http.services.opensearch-secure.loadbalancer.server.port=4180"
        - "traefik.docker.network=reverseproxy"
  dashboards:
    image: opensearchproject/opensearch-dashboards:2
    #ports:
     #- 5601:5601
    environment:
     OPENSEARCH_HOSTS: '["http://opensearch:9200"]' # must be a string with no spaces when specified as an environment variable
     DISABLE_INSTALL_DEMO_CONFIG: "true"
     DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"
    networks:
     - shuffle
networks:
  shuffle:
    driver: overlay
    external: true    
  reverseproxy:
    driver: overlay
    external: true

Setup data locations

This could be made as a command but I like to run first and echo instead of the "mkdir -p" to be sure I didn't made a mistake.

# grep "/mnt/docker_data" /mnt/configuration_data/compose_files/SOAR/shuffle-stack.yml | awk -F "- " '{print $2'} | awk -F ":" '{ system("mkdir -p "$1"") }'

Deploy Shuffle stack

Deploy the Shuffle stack by running docker stack deploy -c <path -to-docker-compose.yml> shuffle

Log into your new instance at any nodes on port https://node:3443.

Note

Have you may have seen, all except database have a replica set to 3. This is done so we don't create a messy mess on DB if writing are done from different shuffle-database container at the same time on the same value which will lead to database corruption. Anyway, this container is up in 7 second in case of failure of the node ^^ .

App gonna be slow to run on it's first usage on all new nodes where Orborus is going to use the base image for the first time. As swarm distribute request around all replicas, sometime request gonna be fast as the node already run the image and sometime not. This all Replica and Clustering of Shuffle is not yet production ready as it is not yet properly implemented.

I recommend using only one database and one Orborus for now.