my gt research: what i'm putting in front of my advisor (week two)
week two of my cs 8903 project at georgia tech: the three results i'm presenting at today's progress meeting, the claims i had to walk back after a hard prior-art search, and the one decision left, which venue to aim for.
week two, in two sentences: ciguard, the filter that strips private data out of the messages ai agents send each other, now has the full picture, it beats every defense i tested, it does not break the task, and it replicates on a benchmark i did not build. today is the progress meeting with my advisor, so this post is the three charts i'm walking in with, plus the parts i had to make more honest along the way.
last week was building the thing. this week was pressure-testing the claims until i trusted them, and getting the story tight enough to defend out loud.
the one chart that matters
this is leakage on the internal channels, the agent-to-agent messages and memory writes, for every defense i could run. lower is better. the redactor agentleak ships (presidio) leaves 40.4% of messages still leaking by meaning. the perimeter tools everyone deploys, llama guard and nemo guardrails, are basically no-ops here, they sit up near 94% because they watch the wrong boundary. ciguard, in green, is at 14.1%, while still keeping about 90% of the fields agents are actually allowed to use. it wins in every domain and on both channels, and the gap clears a paired significance test at p below 1e-82.
the part i like is that the win does not come from one clever trick. the deterministic stage kills the literal forwarding, the rewrite stage kills the paraphrases that a name-and-number redactor leaves behind. take either stage out and the number gets worse.
does it protect by quietly breaking the task? no.
the easy way to look good on a privacy metric is to delete everything and wreck the task. so i ran the agents live, coordinator handing work to a worker, with the gatekeeper off and then on, and measured both leakage and whether the job got done. the red bars are leakage, which collapses. task success barely moves, and in the cross-model setting it actually goes up, because stripping the junk makes the hand-off cleaner. this holds across two different model setups and four domains agentleak never tested.
does it only work on the benchmark i built it on? no.
this is the one i cared about most. i took the exact same gatekeeper, no tuning, and ran it on privacylens, a separate benchmark from a different group, scored by privacylens's own leak metric. to keep myself honest i ran it under two independent judges. leakage drops by half to three-quarters under both, and helpfulness stays put. same code as the live runs, so this is transfer, not refitting.
the part i walked back this week
two things got more honest, and i think the paper is better for it.
first, the metric. measuring leakage by meaning needs a judge you trust, and the benchmark's default judge is not one, i watched it wave through a plain ssn and a diagnosis. so the headline semantic numbers lean on a judge i validated against ground truth, and the load-bearing comparison is agentleak's own exact-match detector, which ciguard also wins decisively. that matters: the "we beat their benchmark" claim does not even depend on my choice of semantic judge. and to be fair to agentleak, anchoring on exact match is the right design, this is a measurement choice on my end, not a knock on them.
second, novelty. i ran a deliberately skeptical prior-art search to try to scoop my own idea, and i found the closest neighbor, abdelnabi et al., a contextual-integrity firewall on agent-to-agent messages. it is real and it is close. but it runs on a different benchmark, uses learned abstraction rules instead of redact-then-rewrite, and does not cover the shared-memory channel. so i rewrote the framing to cite and distinguish it rather than claim to be first. better to find that myself than have a reviewer find it for me.
the one decision today
my advisor's bar is replicate before you build, then exceed what you built on. on that bar this is done: i took a strong recent paper, reproduced it, and beat its baselines on its own metric plus a third-party benchmark. so the question now is where it goes. ieee access is the fast, certain path, and after the prior-art search the novelty reads as solid but incremental, which argues for access over a top security venue. that's the call we make in the meeting.
next
assuming the framing holds up, fold in the results sections, do a full citation pass against the related work i just dug up, add the boring publication scaffolding, and pick the venue for real.
more next week.