Question for you ... How do you define DPI on a mouse or a trackball for that matter?
For me, it's the number of increments the device is able to detect while rolling the ball on one linear inch.
Baud rate is also a factor because if you roll it too fast, you might skip some increments.
I think it would be better to set the trackball at its highest speed using the Kensington drivers/software and then measure the minimum ball mouvement required to make the cursor move.
For the experiment, I would use a properly fixed micrometer to operate the ball
BTW, I think using the lowest resolution possible on the monitor may even help to detect the movement.
Or even better if you are equiped to actually sniff the device output traffic on the RS232/PS2/USB port.
EDIT:
There you go... Just found a USB sniffer