Amartya Gaur
07Agent systems
← The index

Evals with a deploy gate

739 cases that can fail a build.

Made for
Newfold Digital
Years
2024 – 2026
Status
In production

739 regression cases, ~2,700 assertions, ~250 scoring metrics, wired into Jenkins as a blocking gate. Deterministic assertions can fail a build; model-judged ones report and stay out of the way.

Nearly every take-home test ever set is the second version. It looks rigorous and it cannot separate anybody.

Can the marking tell them apart?

Two people hand in the same task. One understood it; one wrote something that looks right. If the marking cannot separate them, the marking is worthless.

Fig. 1

The task: stop a runaway program from calling something ten thousand times90 minutes

Answer A someone who understood the problem

class Limiter:
    def __init__(self, rate, per):
        self._buckets = TTLCache(ttl=per * 2)
        self._lock = asyncio.Lock()

    async def take(self, key):
        async with self._lock:
            b = self._buckets.get(key) or Bucket(rate, per)
            return b.consume(monotonic())

Answer B looks fine, is not

counts = 

def take(key):
    now = time.time()
    window = int(now // 60)
    hits = counts.get((key, window), 0)
    if hits >= 10:
        return False
    counts[(key, window)] = hits + 1
    return True

What the marking checks8 cases

  1. Lets through a normal burst··
  2. Blocks the eleventh request in a minute··
  3. Lets you back in when the minute is up··
  4. Keeps one customer from spending another’s··
  5. Still holds when two people click at once··
  6. Gives nothing away if the clock jumps back··
  7. Stops cleanly when the service shuts down··
  8. Does not grow forever in memory··