CLI Ergonomics

The Goldentooth CLI underwent a fundamental transformation, evolving from a verbose, Ansible-heavy interface into a sleek, ergonomic command suite optimized for both human operators and programmatic consumption. This architectural revolution introduced direct SSH operations, intelligent MOTD systems, distributed computing integration, and performance improvements that deliver 3x faster execution times.

The Transformation

From Ansible-Heavy to SSH-Native Operations

The original CLI relied exclusively on Ansible playbooks for every operation, creating unnecessary overhead for simple tasks. The new architecture introduces direct SSH operations that bypass Ansible entirely for appropriate use cases:

Before: Every command required Ansible overhead

# Old approach - always through Ansible
goldentooth command all "systemctl status consul"  # ~10-15 seconds

After: Direct SSH with intelligent routing

# New approach - direct SSH operations
goldentooth shell bettley                    # Instant interactive session
goldentooth command all "systemctl status consul"  # ~3-5 seconds with parallel

Revolutionary SSH-Based Command Suite

Interactive Shell Sessions

The shell command provides seamless access to cluster nodes with intelligent behavior:

# Single node - direct SSH session with beautiful MOTD
goldentooth shell bettley

# Multiple nodes - broadcast mode with synchronized output
goldentooth shell all

Smart Behavior:

  • Single node: Interactive SSH session with full MOTD display
  • Multiple nodes: Broadcast mode with synchronized command execution
  • Automatic host resolution from Ansible inventory groups

Stream Processing with Pipe

The pipe command transforms stdin into distributed execution:

# Stream commands to multiple nodes
echo "df -h" | goldentooth pipe storage_nodes
echo "systemctl status consul" | goldentooth pipe consul_server

Advanced Features:

  • Comment filtering (lines starting with # are ignored)
  • Empty line skipping for clean script processing
  • Parallel execution across multiple hosts
  • Clean error handling and output formatting

File Transfer with CP

Node-aware file transfer using intuitive syntax:

# Copy from cluster to local
goldentooth cp bettley:/var/log/consul.log ./logs/

# Copy from local to cluster
goldentooth cp ./config.yaml allyrion:/etc/myapp/

# Inter-node transfers
goldentooth cp allyrion:/tmp/data.json bettley:/opt/processing/

Batch Script Execution

Execute shell scripts across the cluster:

# Run maintenance script on storage nodes
goldentooth batch maintenance.sh storage_nodes

# Execute deployment script on all nodes
goldentooth batch deploy.sh all

Multi-line Command Execution

The heredoc command enables complex multi-line operations:

goldentooth heredoc consul_server <<'EOF'
consul kv put config/database/host "db.goldentooth.net"
consul kv put config/database/port "5432"
systemctl reload myapp
EOF

Performance Architecture

GNU Parallel Integration

The CLI intelligently detects and leverages GNU parallel for concurrent operations:

Automatic Parallelization:

  • Single host: Direct SSH connection
  • Multiple hosts: GNU parallel with job control (-j0 for optimal concurrency)
  • Fallback: Sequential execution if parallel unavailable

Performance Improvements:

  • 3x faster execution for multi-host operations
  • Optimal resource utilization across cluster nodes
  • Tagged output for clear host identification

Intelligent SSH Configuration

Optimized SSH behavior for different use cases:

Clean Command Output:

ssh_opts="-T -o StrictHostKeyChecking=no -o LogLevel=ERROR -q"

Features:

  • -T flag disables pseudo-terminal allocation (suppresses MOTD for commands)
  • Error suppression for clean programmatic consumption
  • Connection optimization for repeated operations

MOTD System Overhaul

Visual Node Identification

Each cluster node features unique ASCII art MOTD for instant visual recognition:

Implementation:

  • Node-specific colorized ASCII artwork stored in /etc/motd
  • Beautiful visual identification during interactive SSH sessions
  • SSH PrintMotd yes configuration for proper display

Examples:

  • bettley: Distinctive golden-colored ASCII art design
  • allyrion: Unique visual signature for immediate recognition
  • Each node: Custom artwork matching cluster theme and node personality

Smart MOTD Behavior

The system provides context-appropriate MOTD display:

Interactive Sessions: Full MOTD display with ASCII art Command Execution: Suppressed MOTD for clean output Programmatic Access: No visual interference with data processing

Technical Implementation:

  • Removed complex PAM-based conditional MOTD system
  • Leveraged SSH's built-in PrintMotd behavior
  • Clean separation between interactive and programmatic access

Inventory Integration System

Ansible Group Compatibility

The CLI seamlessly integrates Ansible inventory definitions with SSH operations:

Inventory Parsing:

# parse-inventory.py converts YAML inventory to bash functions
def generate_bash_variables(groups):
    # Creates goldentooth:resolve_hosts() function
    # Generates case statements for each group
    # Maintains compatibility with existing Ansible workflows

Generated Functions:

function goldentooth:resolve_hosts() {
  case "$expression" in
    "consul_server")
      echo "allyrion bettley cargyll"
      ;;
    "storage_nodes")
      echo "jast karstark lipps"
      ;;
    # ... all inventory groups
  esac
}

Installation Integration:

  • Inventory parsing during CLI installation (make install)
  • Automatic generation of /usr/local/bin/goldentooth-inventory.sh
  • Dynamic loading of inventory groups into CLI

Distributed LLaMA Integration

Cross-Platform Compilation

Advanced cross-compilation support for ARM64 distributed computing:

Architecture:

  • x86_64 Velaryon node: Cross-compilation host
  • ARM64 Pi nodes: Deployment targets
  • Automated binary distribution and service management

Commands:

# Model management
goldentooth dllama_download_model meta-llama/Llama-3.2-1B

# Service lifecycle
goldentooth dllama_start_workers
goldentooth dllama_stop

# Cluster status
goldentooth dllama_status

# Distributed inference
goldentooth dllama_inference "Explain quantum computing"

Technical Features:

  • Automatic model download and conversion
  • Distributed worker node management
  • Cross-architecture binary deployment
  • Performance monitoring and status reporting

Command Line Interface Enhancements

Bash Completion System

Comprehensive tab completion for all operations:

Features:

  • Command completion for all CLI functions
  • Host and group name completion
  • Context-aware parameter suggestions
  • Integration with existing shell environments

Error Handling and Output Management

Professional error management with proper stream handling:

Implementation:

  • Error messages directed to stderr
  • Clean stdout for programmatic consumption
  • Consistent exit codes for automation integration
  • Detailed error reporting with actionable suggestions

Help and Documentation

Built-in documentation system:

# List available commands
goldentooth help

# Command-specific help
goldentooth help shell
goldentooth help dllama_inference

# Show available inventory groups
goldentooth list_groups

Integration with Existing Infrastructure

Ansible Compatibility

The new CLI maintains full compatibility with existing Ansible workflows:

Hybrid Approach:

  • SSH operations for simple, fast tasks
  • Ansible playbooks for complex configuration management
  • Seamless switching between approaches based on task requirements

Examples:

# Quick status check - SSH
goldentooth command all "uptime"

# Complex configuration - Ansible
goldentooth setup_consul

Monitoring and Observability

CLI operations integrate with existing monitoring systems:

Features:

  • Command execution logging
  • Performance metrics collection
  • Integration with Prometheus/Grafana monitoring
  • Audit trail for security compliance

User Experience Improvements

Intuitive Command Syntax

Natural, memorable command patterns:

# Intuitive file operations
goldentooth cp source destination

# Clear service management
goldentooth dllama_start_workers

# Obvious interactive access
goldentooth shell hostname