From Noise to Insight: The DoD’s Next Battle with Public Information

The recent Small Wars Journal article, “Beyond Collection: Building Publicly Available Information Systems,” co-authored by Major Iain Cruickshank - a friend of Storm King - makes a compelling case for why militaries can no longer just hoard publicly available information (PAI) - they need to put it to work.

For years, government agencies and the military have gathered PAI from social media, commercial satellites, public databases, and news sources. But as the article points out, most of that data still sits in isolated pockets, disconnected from larger intelligence systems. To really tap its value, the Department of Defense (DoD) needs to move beyond collection and build enterprise-level systems that can ingest, process, and share PAI across the force.

The article’s core message is simple but powerful: PAI isn’t just a niche supplement to classified intel anymore—it’s often just as valuable, sometimes even more so. Whether it’s for early warning, situational awareness, or geopolitical insight, PAI can help the DoD make faster, smarter decisions. But doing that at scale requires automation, smart architecture, and seamless integration across classified and unclassified domains.

That’s where Retrieval-Augmented Generation (RAG)-AI systems come in. These tools combine large language models with live access to external data, helping analysts quickly pull in relevant, real-time info and turn it into usable insights. It’s a way to cut through the noise and help commanders act faster.

But here’s the catch: turning this vision into reality inside the DoD isn’t as easy as flipping a switch.

Challenges and Other Solutions

First, there’s the cybersecurity elephant in the room. Bringing open-source pipelines into secure DoD networks creates fresh attack surfaces. Adversaries could inject bad data or exploit vulnerabilities, potentially corrupting what gets fed into models. RAG-AI systems that draw from the web are especially at risk without proper safeguards, testing, and isolation.

Second, data governance gets tricky. The DoD needs to know where its data comes from and how it was processed. Without clear accountability, leaders could end up making critical decisions based on flawed or manipulated inputs.

Third, there’s an institutional challenge. Classified environments are understandably cautious about blending unclassified sources with sensitive workflows. It’s going to take more than just shiny new tech - it’ll require new policies, cultural shifts, and careful change management.

And finally, let’s not forget the infrastructure gap. Many deployed environments lack the bandwidth or computing power to run cutting-edge AI at the edge, meaning there’s a real risk of leaving units behind.

It’s worth noting, though, that RAG-AI isn’t the only game in town. Traditional OSINT platforms like Palantir or Recorded Future, human-machine teaming models, edge processing, knowledge graphs, synthetic data for training, and smarter partnerships with allies or commercial providers all offer pieces of the solution. A mix-and-match approach may ultimately work best, balancing automation with human judgment, and speed with security.

Importantly, these challenges don’t just apply to the U.S. military. Commercial organizations and international security partners face similar issues. Companies working in critical infrastructure, finance, or crisis response increasingly rely on PAI, and they need to manage cyber risks, data governance, and ethical use just as carefully. For anyone operating in today’s data-saturated world, the question isn’t just can we access the data, but how we do it safely, ethically, and effectively.

Bottom line: the article nails the why of PAI integration, but the how, especially inside a complex organization like the DoD, requires careful attention to cyber risk, governance, and culture. If done right, this shift could give both military and civilian organizations a major edge in a fast-moving, data-driven world.

As organizations across sectors wrestle with the promise and pitfalls of PAI, now is the time to invest in the right mix of technology, governance, and human expertise. Whether you’re in defense, industry, or international security, the challenge is clear: turn data into insight without compromising trust or security. At Storm King, we’re ready to help leaders navigate this transformation and build systems that don’t just collect data, but make it count. Let’s get to work.