2025-03-28 | | Total: 2
We present a convex optimization framework for overcoming the limitations of Kubernetes Cluster Autoscaler by intelligently allocating diverse cloud resources while minimizing costs and fragmentation. Current Kubernetes scaling mechanisms are restricted to homogeneous scaling of existing node types, limiting cost-performance optimization possibilities. Our matrix-based model captures resource demands, costs, and capacity constraints in a unified mathematical framework. A key contribution is our logarithmic approximation to the indicator function, which enables dynamic node type selection while maintaining problem convexity. Our approach balances cost optimization with operational complexity through interior-point methods. Experiments with real-world Kubernetes workloads demonstrate reduced costs and improved resource utilization compared to conventional Cluster Autoscaler strategies that can only scale up or down existing node pools.
In this work, we propose a metric called Number of Thoughts (NofT) to determine the difficulty of tasks pre-prompting and support Large Language Models (LLMs) in production contexts. By setting thresholds based on the number of thoughts, this metric can discern the difficulty of prompts and support more effective prompt routing. A 2% decrease in latency is achieved when routing prompts from the MathInstruct dataset through quantized, distilled versions of Deepseek with 1.7 billion, 7 billion, and 14 billion parameters. Moreover, this metric can be used to detect adversarial prompts used in prompt injection attacks with high efficacy. The Number of Thoughts can inform a classifier that achieves 95% accuracy in adversarial prompt detection. Our experiments ad datasets used are available on our GitHub page: https://github.com/rymarinelli/Number_Of_Thoughts/tree/main.