Hi, I have a question regarding the termination condition for each batch.
longspec_benchmark.py terminates a batch if a single instance in the batch reaches desired amount or produces eos token.
But shouldn't the system wait for all other instances to finish?
Or is this a technique used commonly to estimate performance of speculatvie decoding?
Hi, I have a question regarding the termination condition for each batch.
longspec_benchmark.py terminates a batch if a single instance in the batch reaches desired amount or produces eos token.
But shouldn't the system wait for all other instances to finish?
Or is this a technique used commonly to estimate performance of speculatvie decoding?